What are the prospects for self-sustained impact at the Macro level? (Part 1)
After studying various impact evaluations, it became clear that different development projects also have different scales, with multilaterally-funded macro-level being focused on higher scale objectives from the municipal up to the national level while micro-level are geared towards improving socioeconomic aspects of communities. Our research revealed that there is a recurring error among projects that makes them ineffective: a lack of community involvement in all stages of the project, or more specifically, their lack of involvement in both the design and the evaluation process. This results in a lack of opinions from the local level regarding project self-sustainable goals, and prospects for ultimate success in self-sustainability. We have seen both macro projects, implemented by multilaterals, and micro projects, implemented by international non-profits, that fail to include local participation in their methodology, but one trend we have noticed is that projects on the macro level, have a tendency to plan, implement, and complete projects with little input from local participants or consideration for the possibility of self-sustainability impacts.
A particularly egregious example of non-self-sustainability at any level is the project for OECD-funded Improving the Performances and Management of Public Lighting in Ho Chi Minh City, which was a project with no ground level/community participation, but rather took place at the municipal level between French company Citelum (supplying technical assistance and equipment) and Sapulico (Saigon Public Lighting Company, Ho Chi Minh City's public lighting authority). From the evaluation summary, we saw the project was most concerned with propagating appreciation for French expertise in the area of public lighting at the national level rather than fostering national electrical self-sustainability. With organizational sustainability as a main objective, the project itself was not considered self-sustainable for a few key reasons:
“Sapulico's engineers acquired the necessary [skills required to operate the system], however, their current skill level is insufficient to handle present and future technical difficulties without the assistance of Citelum.” This means that the local company in Vietnam would continue to be reliant on foreign expertise in the likely event of future technical difficulties.
Also, jarringly, “there are also concerns even about the project's current viability as Sapulico's budget doesn't allow for the purchase of equipment from abroad, therefore the company will be unable to replace equipment that has reached the end of its lifecycle.”
This highlights an obvious overlooking of future project sustainability due to the local budget conditions and insufficient training. Had the project been more concerned with guaranteeing even national self-sustainability rather than the financial sustainability of the foreign implementing organization, the project overall could have been more successful far into the future….
A more promising example of a macro level project aiming for more localized impact, yet still failing to consider self-sustainability, is the JICA Agricultural Development Project in Kambia District. While we commend JICA as the most active multilateral to do ex-post evaluation, we again see the tendency of macro level projects to focus on systemic changes at the national or at least regional level which often self-sustainability buy-in, whereas micro level projects tend to focus on communities or at most sub-regions with the objective of fostering longer-term sustainability. This project, on the other hand, is a hybrid of macro- and micro- as it was to intervene at a district level and it did try to create agricultural extension buy-in. The key takeaways from JICAs post-project evaluation:
This project was concerned with improving agricultural productivity at select sub-district pilot sites, and then a further extension of the project throughout the entire Kambia District. The evaluation focuses heavily on the inability of the project to complete this objective of extending the project geographically because there was insufficient manpower, technical support, and funding to complete project dissemination, which suggests that sustained impact was not fully considered when planning implementation. It also suggests it did not expand because lessons seem not to have been learned from the implementation, and funding was not longer available.
The key point here is that the project did not secure adequate resources or funding for the true completion of its objective, which was to extend the successful outcomes that they observed at the pilot sites further across the region. This resulted in an evaluation analysis of sustainability that was only “fair”.
This project also fails to mention any local participation in the design, planning, implementation, or long-term sustainability phases, so despite seeing successful outcomes in increased crop production at the pilot locations, the evaluation was ultimately negative.
It is clear from the evaluation that the project was being executed from the side of the implementing agency rather than involving local participation beyond the extensionists, and no input was asked from community members about resulting agricultural self-sustainability.
The crux is that had the project been more focused on local self-sustainability beyond the regional extension system, its own organizational deficiencies (lack of manpower, technical support, and funding mentioned above) might not have hindered the ability to expand the project further.
Compare these findings to those we have analyzed in other blogs which illustrated the far more successful project and evaluation results that micro level non-profit development agencies like Mercy Corps, Plan International, and Partners for Democratic Change fostered when they made community participation the key to their development process. However, JICAs Kambia project and the Vietnamese Public Lighting program are not alone in their shortcomings, because the consistent lack of local community involvement in community-focused projects is a theme that we continue to see with macro level and micro level development schemes – a pattern that Valuing Voices is determined to change.
Overall, what we have discerned from this analysis is that projects conducted by macro level multilateral actors have a lower chance of ensuring self-sustainability, in part because they usually aren’t linked directly to the community in regional or national-scale projects. By contrast, micro level projects by international non-profits tend to have a higher chance of fostering greater self-sustainability because they are linked more closely to the end users of development, thereby benefiting both the organization itself as well as the local participants from more inclusionary practices.
We can clearly see that at least a few organizations, both at the macro and micro level, are asking the right questions by conducting these post-project evaluations. However, a systemic problem persists in that virtually no one is designing for self-sustainability. But without it, how successful are we really?
Twigas (Participant self-sustainability) throughout 'development time'- Design, Implementation, M&E and beyond
What if we saw our true clients (sometimes vulnerable twigas) as our project participants and wanted the return on investment of projects be maximally sustained? How would this change how we design, implement, monitor and evaluate projects with country-leadership? With $1.5 trillion dollars in US and EU foreign aid spent since 2000, don't you think that our industry urgently needs feedback on what communities feel will be sustainable now, what interventions offer the likelihood of positive impact beyond the performance of the project’s planned (log-framed) activities, and what activities stood the test of time and our departure?
Shockingly, such learning is not happening today. YET.
Country-led development through participation has been a clarion call for several years, including USAID Forward and EU’s interest in food security and development finance. The Paris Declaration on Aid Effectiveness is a clarion call supporting country-led development. Multilaterals do some work to evaluate their projects; a handful of international non-profits do a small amount of evaluation post-project; and non-profits do some work using mobile technology to harness community feedback. Africa’s CAADP also manifests country-led and partially country-financed development (5-10% of GDP budgeted for Agriculture). Aid transparency is gaining support, and ValuingVoices adds a much needed local voice in terms of transparent, effective assistance. Much more funding is emerging to foster grassroots participation (e.g., Ashoka, Rockefeller and other funding), ranging from lessons learned from 6,000 aid recipients by Time to Listen to books touting Participatory Statistics, and the astonishing growth of mobile crowdsourcing platforms such as Kenya’s Ushahidi and Ghana’s Esoko. IOCE’s new evaluation initiative (VOPE) that is building capacity of national evaluators will also be a key resource.
So that's the good news coming up the riverbank. When should we be talking to communities about their views of what they feel is most sustainable and that they can sustain themselves? ALL THE TIME.
1) During Design
Most of our projects are designed in capital cities by donors who are informed by past projects, research, industry trends, policy imperatives, and are limited by narrowly focused funding streams. Very few projects are designed by those who will live with the results, namely the communities themselves. Donors often develop proposal requirements with little input from locals until start-up, when implementers fine-tune. Communities only get what donors are willing to give and donors want results. Not just temporary results but sustained ones. Such faulty design and lack of learning from evaluations—rarely asking our true clients, our participants, from the very start—leads to rusting tractors, farmers returning to traditional practices without resources to switch, latrines that are not used, etc. We mean well, but these practices are not working well. We know this from working over 25 years in international development. We need to change that.
2) During Implementation, at Midterm and Final Evaluation
Communities participate in midterm and final evaluations but often the knowledge is a one-way street, out. Hardly ever are findings shared with communities and joint re-design done to optimize prospects for self-sustainability of project activities. Ongoing learning from implementation needs to be transparently captured and shared in open-data format for discussion and adaptation (along USAID’s CLA Approach – Collaborating, Learning and Adapting program activities). Often sustainability is planned during the last 6 months of implementation, and handed to partners that are weakly-resourced, may contain untested assumptions, and haven’t had a chance to build up the organizational culture needed. Capacity building of partners to take over implementation also requires resources, which need to be budgeted in, and systems that partners co-create to fit their needs and ability to maintain. Program evaluation data that exists is often locked away in .pdf files in repositories that are not often accessible by countries themselves, and lessons learned within doesn’t often inform future design even among the implementers, much less by locals. We must change that.
3) After Project Close-out
Very rarely do implementing agencies return 3, 5, 10 years after projects close and ask participants what is “still standing” that they managed to sustain themselves. How often do we take community members, local NGOs, national evaluators as the leaders of evaluations of long-term self-sustainability of our projects? Based on my research (www.ValuingVoices/blogs) 99% of international aid projects are not evaluated for sustainability or impact after project close by anyone, much less by the communities they are designed to serve. We can change that.
This is what ValuingVoices is all about.
'Developing them' fails (Haiti). Long live country-led development! (?)
I am tired of inefficiency and waste of tax dollars and human capacity and time. Some organizations use images of Africans, Asians, Latin Americans and Caribbean people suffering without permission to further their mission to get funds to do programming. United for Sight says it "evokes the idea that the poor are helpless and incapable of helping themselves, thereby cultivating a culture of paternalism." And that's what I am the very most tired of, our paternalism; this is the antithesis of country-led development.
Maybe we've gone about it a bit wrong. A case in point is a recent report on Haiti, that the US spent $2.8 billion and that there is little to show for it. Now, emergency response is truly a different kettle of fish – it is very expensive to intervene and restore even basic services and many nations have low aid-absorptive capacity. Nonetheless, it opens the door to unbridled waste, low capacity building and little sustainability as well. As Global Post and AP found, only 5% of 2012 US government spending went through Haitian organizations (and 95% through US and others) so the following dismal results are on our plate:
* "The number of people still living in grim encampments of quake survivors is now at 170,000" and while "The US promised to build 15,000 permanent homes but completed only 2,649 of them before ending its housing construction program, deciding instead to extend financing to Haitians directly for them to build their own homes." (I think at least we've tried to let people get in debt and find their own way to house themselves :), like us)
* The US is spending $170 million "to attract manufacturing to the new Caracol Industrial Park in Haiti’s North. But the massive investment was to have created 65, 000 jobs but has created fewer than 3,000 jobs, and the project may not have the funds to construct the port needed to export the industrial goods." Further, a US 2013 GAO audit found that there remained a deficit of over $100 million to complete it, with questionable prospects of success. (Not the best Return on Investment)
* “If the dismal record of USAID’s most trusted contractor is any indication, these entities are unreliable to say the least. A government audit of Haiti projects performed by USAID’s largest contractor, Chemonics, shows the company routinely failed to implement its aid projects correctly. In 2012, GlobalPost discovered that a $2-million, US-taxpayer funded USAID/Chemonics project constructed a building for Haiti’s parliament that was unfinished and unusable.”
As one Haitian professional said, "“There have been aid programs for such a long time here, but when you evaluate it, they don’t have durable impacts… [it] isn’t really helping the people with these problems be released from their problems, it only keeps them stuck in poverty.”
So this is what can happen when we know best. Again and again, with great hubris and waste, we know better than the country's nationals. I have worked in "international development" for 25 years and the title has always irked me. Are international development projects named aptly? Isn't it instead a matter of national development projects supported by international expertise and funding instead?!
There is good news. USAID recently launched its Forward Progress program, which aims to increase the amount of money it sends directly to local companies and NGOs (Congress permitting). "we have to support the institutions, private sector partners and civil society organizations that serve as engines of growth and progress for their own nations. USAID Forward is helping us to do that through new models for public-private partnerships and increased investment directly to partner governments and local organizations." The jury's still out about tangible results locally, but this sounds hopeful.
A delightful prospect of real country-led development potentially comes from a collaborative international donor-country partnership, CAADP. Comprehensive Africa Agriculture Development Programme's (CAADP) goal is to eliminate hunger and reduce poverty through agriculture by addressing "policy and capacity issues across the entire agricultural sector and African continent. CAADP is entirely African-led and African-owned and represents African leaders' collective vision for agriculture in Africa." Notably, putting their money where their mouths are, "African governments have agreed to increase public investment in agriculture by a minimum of 10 per cent of their national budgets and to raise agricultural productivity by at least 6 per cent."
So how are is CAADP doing after 10 years? The results are mixed. A 2013 IFPRI evaluation found that
* "Unlike programs that are largely driven by donor priorities, undermining local institutions in the process, CAADP is a product of the locally-sourced development goals of participating countries. This approach has resulted in a sense of ownership and better development outcomes" [yet]
* “The countries with the 10 largest agricultural sectors on the continent… have spent less than 5 percent of their total budget"
But while these results are mixed, so too are donor commitments. The 2011 midterm evaluation found that while there were compacts created in 22 countries, and 6 countries have been awarded GASFP funding totaling $270million, another 8 countries across West Africa (plus Ethiopia) received less than 50% of the funds promised to promote agriculture.
So what will it take for the poor to get the help they need from any of us? Do we value their voices? Is our international aid really helping them? You tell me.
Let's start turning the oceanliner of development to dignified sustainability – today!
No time like the present, our participants are waiting for dignified development to fully arrive. Dignity is the "quality or state of being worthy of esteem or respect." When we design projects with communities with long-term self-sustainability as the core value, we are respecting them.
And we must start local. As Acumen Fellow Natalie Grillon says from Uganda: "Ask more questions, listen and learn. I’m always trying to get better at listening to learn before I act so that my actions can lead to productive results based on consensus and conversations rather than assumptions. Don’t believe that you know the full story. The farmers for whom I work know their crop, know their land and know what they want."
She goes on to tell us how she stopped doing things for the project staff: " I could offer more in working alongside my colleagues to learn how to do new things together, like using a new database or thinking about ways to motivate teams… I could feel the sense of empowerment and excitement that could come from learning even a simple new thing and which only required a few minutes of our time" rather than just doing it herself for them.
This is what Valuing Voices is all about — seeing our participants as our true clients, as listening to what they want rather than designing development requests for proposal and even project proposals themselves in capital cities far from input by these very clients. Would any corporation in their right mind design without asking potential clients, without pre-testing and tweaking interventions before a full launch? Imagine how products would fare without marketing campaigns touting their benefits over other products? This doesn't happen often in development because participants are most often seen as 'price takers', grateful for what we offer, not matter if the fit is not perfect. Such an approach is not only inefficient, it is deeply disrespectful.
On the other hand, there are successes to be seen along the lines of localizing development. In addition to successes found in my other blogs (see Plan, Mercy Corps, LWR and even PACT – forthcoming), Devex recently posted an article praising Nuru and Millenium Villages for "working to develop local leaders who will fully take over the various programs they’ve begun after the foreigners leave in two years… [using] the opportunity to build up local leaders and engage them in developing their own communities." Hallelujah! Devex says Munk's critique of Millenium Villages is that "Millennium violates two basic principles of good development: It’s not scalable and it’s not sustainable". Nuru believes "poor people hold the keys to their own development."
Absolutely! Not only should design and implementation center around community wants and capacities, there are all sorts of project activities that communities can help sustain themselves:
* Agriculture and livelihoods (income generation, micro-credit, marketing),
* Natural resource management (climate-smart agriculture, reforestation),
* Literacy and numeracy.
While there are other things larger than communities that need ongoing external support for, how often are there referendums on what they would prioritize? The UN recently named 2014-2024 the sustainable energy decade and infrastructure such as roads, water systems as well as trade including the World Trade Organization's Doha Trade talks for improve the trading prospects of developing countries are vital, how often have citizens been asked? Ashoka's Changemakers supports projects that create feedback loops (like our own in East Africa: http://www.changemakers.com/project/valuing-voices-kenya).
Communities are deeply grateful for assistance yet they want to to have a voice, to steer the ship more themselves. The Listening Project found that "agencies should slow down and take more time to understand people’s capabilities, priorities, preferences, and ideas… [participants] don’t want to have aid agencies to be more extractive in how they gather information. They want to be part of the decision making process of aid efforts. This goes beyond two-way communication and requires rethinking many of our assumptions and processes to find ways to truly collaborate and support those who are affected."
Among the project's 6,000 interviewees, some wondered "why no one seems to check on whether the assistance provided has made a positive difference in recipients’ lives. It is important for aid agencies to have processes and mechanisms to receive and provide feedback to communities and to be accountable for their actions—and particularly for any harm that has been done."
Yes! We need to help those projects such as Nuru whose farmers are supported in self-sufficiency, countries that support communities evaluating our projects, for a start. That is the dignified life path to take as a development professional – as their peers.
Development= A Jeep (motor optional).. Resilience? If within 5 years!
Imagine being given a lovely new Jeep. You get a driver (remember driving school) to help you learn to steer it around the pothole-strewn, scantly lit roads. Eventually you take over the controls of the Jeep and control the steering wheel directly, driving offroad, with the copilot praising your good driving and steering only to avoid catastrophe. You are told that one day the Jeep will be yours.
That day arrives. The development agency hands you the keys to the Jeep. You wave good bye to them, return to the Jeep, turn the key. Dead.
Looking under the hood, you realize the motor is gone. Checking the rest of the Jeep you realize there is no fuel and the tries are flat. That is what it is like from the community's view of development projects after close-out. The local NGO to whom the project has been 'handed over' has scant financial or human resources to continue (no engine), and in the last few months' scramble to close out, the implemeneting agency put in few systems for communities to continue doing the programming without support by the local NGO or all the resources they had poured in (no fuel). There is little to help you move the Jeep (even on flat tires) except your own feet, other than the capacity building that was learned early on, as it wasn't built to last based on local materials. Sustainability isn't programmed in projects that have set timelines and donor-set markers of success which mandate close-out.
So you own the Jeep but with little power to move, very much like countless well-meant tractor for development agriculture before you.
There are several glimmers of hope. What communities have is the human power that exists locally, fuelled by participation coupled with information transmission such as WorkWithUs and MakingAllVoicesCount (based on the moral imperative of it's Their Development as well) and ALNAP's push to use evaluation for learning in international development.
We could also help donors springboard to learn from sustainability via the latest buzzword in development: "Resilience". At USAID and DIFD this is the "ability of people, households, communities, countries, and systems to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth" and "helping communities and countries to be better prepared to withstand and rapidly recover from a shock", respectively.
Resilience could be the doorway to getting community-defined sustainable programming to break the cycle of recurrent emergencies that divert resources from long-term development. Imagine: we could ask citizens what will make them resilient! A rare, shining example is a USAID-funded Ethiopia project with a mandate to use participatory impact assessments (process monitoring plus participatory input to capture local perceptions of benefits) to learn from communities. A USAID Solicitation tells us "seventeen impact assessments on different program activities were undertaken to inform best practice and to develop guidelines and policies. A major impact was the development and adoption of Emergency Livestock Guidelines by the Ethiopian government. These were based on best practice assessments in many countries (including Kenya) and action research on different types of interventions. Emergency de-stocking–selling livestock early in a drought to preserve their price and leave more fodder and water for remaining animals — was found to be particularly effective, with a 40:1 benefit cost ratio. Emergency livestock vaccination campaigns, on the other hand, were found to have no impact on livestock mortality, and were dropped in favor of other health interventions including parasite control and de-stocking."
Excellent Valuing of pastoralist Voices! How are such locally-informed excellent processes and findings being widely shared and implemented? What do you think?
The 'causes and conditions' are right for sustainable impact… $100 billion of funding in 2014 alone needs it!
Many of us know that when the time is right, things click into place and manifest, but when they are not yet ripe for change, they won't. It is time to evaluate the sustainabilty of development programs; $20 billion of US assistance in 2014 and $80 billion of EU program assistance in 2014 alone. Valuing participants voices for how sustainably this is spent depends on us.
The Paris Declaration and Accra Agenda for Action are powerful statements by donor countries and recipient countries to share responsiblities as one inter-ministerial coordination site says:
Country-led development is the responsibility of both countries and development
partners. A country, through its senior government leaders, and with input from development partners, civil society and other constituents, must articulate sound policies and advocate for them to become common priorities. Development partners must be willing to listen to and support those priorities. If both parties take a shared role and hold one another accountable, then a relationship built on trust emerges and the outcome is something everyone wants…"
Were the conditions not right for country-led development to appear, the push for strengthening national Monitoring and Evaluation Systems and national evaluator capacity building could not have appeared. Had we not reached a capacity to evaluate at least some projects with a deep desire to understand our impact, our community could not have begun powerful initiatives aimed at understanding impact evaluation – such as multiple-donor-suppored 3ie(albeit focusing on looking for impact within the 'box' of what we expect to see from our project activities). Had funding not appeared to manifest Paris through initiatives such as the New Partnership for Africa's Development (see blog on incentives) and research called Time to Listen (see blog on grassroots evaluation), we could not begin to see paticipants as our true clients and real partners for great programming.
Finally, were these all not present, I could not have put myself into the shoes of the Malian woman, looking across the breadth of our 52 year-long-lives. I imagined her seeing projects coming and going from her community, most doing good for the years they were there, but with none of them there long enough, or being designed to achieve what communities felt the sustained impact their projects should have. Until now, few asked her opinion on what activities helped her most and designed flexible-implementation projects that she kept informing with feedback based on what worked best amidst change. No one had asked her and her community to evaluate projects themselves, and tell us what they wanted, what had made them the most resilient and 'developed' to never need our assistance again. We can and must value the voices, ask 'her' during current implementation, at close out and long after projects end. The causes and conditions are right. Now.