by Jindra Cekan | Feb 16, 2025 | Aid effectiveness, climate change, Donors, Evaluation, ex-post evaluation, Health programs, Impact Investors, international development, locally-led development, Monitoring and Evaluation, post-project evaluation, Sustainable development, USAID
The draconian cuts to USAID in February 2025 ($40 billion in currently allocated aid assistance, $63 billion in 2024, $71 billion in 2023, Pew) to 120 countries around the globe) are said to address waste and fraud and re-align priorities. Elon Musk of ‘DOGE‘ (government efficiency group tasked by US President Trump, without an approved remit or security clearances) called the Agency “a criminal organization. Time for it to die” (CBS) without citing proof. While aid is currently frozen under Secretary Rubio, the agency will likely be diminished to only humanitarian aid, leaving development programming shuttered. 10,000 USAID staff, thousands of US-funded contractors, and hundreds of thousands of INGO and NGO national staff and participants and ‘beneficiaries’ and partner organizations worldwide are abruptly without funding, inputs (medicines, seeds, research, scholarships, collaborations, etc) or information about timelines or recourse. While judges rule, suffering has already commenced, value chains have been broken, with aid rotting in warehouses, causing losses to donors (un-paid US farmers or contracted firms) and recipients.
FINANCIAL AND ECONOMIC LOSSES
From an ex-post sustainability view, what is difficult is that such sudden closures mean that not only this year’s already approved funding has been frozen, but all projects and programs that were implemented over a typical 1-5 years could not close properly and that received new funding this cycle. This means projects that began from 2019-2025 have not had time to finish implementing much less prepare for exit, nor learn or share lessons about what worked, what did not, and what the world can learn from such implementation. For instance, in an agriculture project, farmers are trained in year 1 of a 5 year project, inputs are given in year 2, with further inputs, farming practices and trainings and linkages with Ministry of agriculture are fine-tuned (e.g. in response to climatic shocks or changes in economic systems) in years 3 and 4, pre-closure in year 5. Some projects such as those in climate funds install the infrastructure by the end of the project, so any earlier closure is even more catastrophically unfinished. So too are capacity development and ongoing health and education and humanitarian help. While many of this current allocation cycle are likely to be new, many can be continuations of older projects. Premature exit dooms most projects that haven’t finished implementing fully or that needed extended funding to complete. As we shared in our nine checklists (see Capacities book chapter) drawing on best practices from sustained exit and ex-post research, sustainability pre-closure takes time. This time is to foster sustained handover to other partners, or finding alternate resources, including making sure capacities will continue to be passed on. Without time to do this plus evaluations and administrative close-out, far more than ‘just’ current billions allocated by Congress for current funding for programming is at stake; investments over the last 5+ years are also at stake. Thus, abrupt closeout means losing Years of funded implementation benefits and lessons.
While less than 1% of global development projects have been evaluated for sustainability, those ex-post studies 2-30 years post-closure have shown that over 50% of all interventions continue years after closure especially those with strong national ownership (see catalytic evaluations done by 40 organizations including USAID). While half-sustained may seem low, these can be successes given the withdrawal of funding, partnering, and capacities especially given strained conditions in-country to continue. Sometimes activities cannot or should not continue as conditions change, but others remain resilient even to climate shocks. One of the great joys of sustainability evaluations has been documenting emering outcomes, where local communities and partners have sustained results in new ways, which projects could learn from for future designs. But we need to complete programs fully and exit designing for sustainability, plus access other evaluations to learn this.
From 2018-2022 (the most recent, similar timeframe of data from the OECD), a total of 841 trillion in official Overseas Development Assistance (ODA) has been disbursed, with the US government’s share of ODA over those years was 51 billion, of which only US “Core support to NGOs, other private bodies, PPPs & research institutes” constituted ‘only‘ 935 million. It is, therefore, unsurprising the effect that withdrawing such amounts of foreign aid plays in the poorest recipient countries (CGD):
- “Eight low-income countries and eight lower-middle-income countries face losing over a fifth of the total foreign assistance they receive (South Sudan, Somalia, Democratic Republic of Congo, Liberia, Afghanistan, Sudan, Uganda, and Ethiopia). For these 8 countries: The economies of these eight low-income countries are so small that aid makes up an average of 11 percent of their total income (based on available GNI data for seven countries). With USAID providing 30 percent of that support, the freeze could create a shortfall equivalent to over 3 percent of GNI—a potentially major economic shock for countries that are home to 410 million people.”
- The economies of these eight low-income countries are so small that aid makes up an average of 11 percent of their total income (based on available GNI data for seven countries). With USAID providing 30 percent of that support, the freeze could create a shortfall equivalent to over 3 percent of GNI—a potentially major economic shock for countries that are home to 410 million people.
- A further eight [countries] rely on USAID for a fifth of their support: Haiti, Lesotho, Zimbabwe, Kenya, Honduras, Angola, Jordan, and Eswatini.
- If USAID’s freeze went on for a year it would leave twenty-three economies facing at least a 1 percent economic shock (sixteen LICs and seven LMICs). Eight of these would face [a 3-9] percent blow: South Sudan (9 percent), Somalia (9 percent), Afghanistan (7 percent), Liberia (4 percent), Syria (5 percent), the Central African Republic (4 percent), Yemen (4 percent), and Micronesia (3 percent).”
These ‘hits’ to the economic system of these countries will cause even more widespread suffering than ‘just’ those directly affected such as HIV+ babies, input-deprived farmers, students, and many more. Further other donors are also cutting budgets, e.g. the Netherlands and now Denmark, following the UK several years ago.
LEARNING LOSSES
Also, information, the vast troves of 60+ years of development experience housed in the USAID DEClearninghouse has disappeared as well. This included over 3 million mentions of evaluation, including over 50 ex-post studies and evaluations, and hundreds of millions of pages of research and programming, projects, and studies across economic development, health, refugees, climate, education, orphans and vulnerable children, democracy, etc. We can now only access a very partial set of archived documents until 2021. Many of us researchers who used USAID’s documents are having to go back to the archives and recover what we can – while it still exists. The loss of learning for researchers is exacerbated by losses of complementary global health data cand climate data (e.g. avian flu, ebola and NOAA climate models no longer funded to share data, etc). How can aid be effective without learning from the past for the future. Given we are approaching climate tipping points, we cannot afford to lose resources, data or focus. Further, we are losing all the institutional and technical knowledge of the staff and contractors being let go.
NEXT?
Other than Reinstatement, it is time for their own governments, wealthy country nationals, other countries, multilaterals, philanthropic foundations, and private donors, large and small, need to step back in (as during the first Trump administration to fill shortfalls). Droves of private ‘impact’ investors would be welcomed with open arms to finally engage in these countries to fill the gap. We can see that conserving democracy and helping the poorest is no longer the governments’ problem around the world. It is in our hands to save it. We can also make sustainable development better: fostering accountability from us to our development partners.
But this time it needs to be in greater collaboration with country national and civic leaders to invest sustainably in sectors and industries they believe have the greatest unmet needs, combined with the greatest prospects of locally-driven development and sustainability. This includes addressing power inbalances, long-term resources, partnerships, and capacities and fostering ownership from the onset and prevent future governments globally from shutting down their assistance without referendums.
What are your thoughts?
by Jindra Cekan | May 16, 2024 | 3ie, Accountability, Adaptation Fund, Aid effectiveness, American Evaluation Association (AEA), Better Evaluation, Catholic Relief Services (CRS), Donors, Evaluation, ex post evaluation, ex-post evaluation, locally-led development, OECD, Participants, post project evaluation, post-project evaluation, SEIE, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation
Grow the .002% of all global development projects that are evaluated ex-post closure for sustainability
It seems like ‘fake news’ that after decades of global development so few evaluations would have peered back in time to see what was sustained. While I was consulting to the Policy Planning and Learning Bureau at USAID, I asked the head of this M&E department who does ex-post sustainability evaluation as I knew USAID had done some in the 1980s, Cindy Clapp-Wincek answered ‘No one, there are no incentives to do it.’ (She later became our advisor.)
Disbelieving, I did a year of secondary keyword research before devoting my professional consulting life to advocating for and doing ex-post evaluations of sustained outcomes and impacts. I searched USAID, OECD, and other bilateral and later multilateral donors’ databases and found thousands of studies, most of which were inaccurately named ‘ex-post’ or ‘post-closure’ studies. Some of the roughly 1,000 projects I looked at at USAID and OECD that came up under ‘ex-post’, ‘ex post’, ‘post closure’ were final evaluations that were slightly delayed, a few were evaluations that were at least one year after closure, but were desk studies without interviews. Surprisingly, the vast majority of final evaluations found were those that only recommended ex-post evaluation several years later to confirm projected sustainability.
In 2016 at the American Evaluation Association conference, a group of us did a presentation. In it, I cited these statistics from of 1st year of Valuing Voices’ research:
- Of 900+ “ex-post” “ex post” “post closure” documents in USAID’s DEC database, there were only 12 actual post-project evaluations with fieldwork have been done in the last 20 years
- Of 12,000 World Bank projects – only 33 post-project evaluations asked ‘stakeholders’ for input, and only 3 showed clearly they talked to participants
- In 2010 Asian Development Bank conducted 491 desk reviews of completed projects, and returned to 18 actual field-based post-project evaluations that included participant voices; they have done only this 1 study.
- We found no evaluations by recipient governments of aid projects’ sustainability
12 years of research, advocacy and fieldwork later, the ‘catalysts’ database on Valuing Voices now shows actual fieldwork-informed evaluations by 40 organizations that had actual ex-posts that returned to the field to ask participants and project partners what was sustained, highlighting 92 ex-posts.
- Ten were multilaterals, international and UN organizations (e.g. World Bank including the Adaptation Fund, OECD, ADB but no IADB or African Development Bank)
- 14 were bilaterals (USAID, GTZ, only a handful from the Nordics and the Netherlands where we did a deep dive into five countries that claimed 32 ex-posts but have only done 16), one from the UK, and surprisingly none were found from France Development, Canada’s CIDA/ Global Affairs, Sweden or Denmark’s governments)
- 16 were non-profits. Notably in the industry, many point to the World Bank as the leader, but after meeting the head of M&E at the IEG in 2015, most have shown to be internal studies, and desk studies or, worse World Bank’s GEF used projected sustainability estimates and claimed actual sustainability of results.
- Notably, JICA is the superstar learning organization in ex-post evaluation, having done more than 2.500 evaluations since 1993 exploring ‘sustained net benefits over time’ of their grants, loans, and technical cooperation. They return three years post-closure and sometimes again seven years post-closure, an amazing feat. Whether their projects are more sustained, as a result is an unanswered question, but at least it’s more likely given they’re learning what worked locally after they went home, what did not, and why and whether their suggested remedies after 3 years were carried out (which they check 7 years post closure).
- Famed evaluator Michael Scriven’s Faster Forward Fund gave us a grant to evaluate eight ex-posts and share methodological highlights; we created an evaluability checklist which readers might find useful.
- I also looked deeply into ‘impact evaluations’ at 3ie, but these turned out to be effectiveness reviews during – not after – implementation. And there are other indications we don’t do quality impact research. “[USAID] agency gives out billions to programs that don’t achieve their intended expectations, and, worse, it’s not even sure of the impact of most of the money it gives in aid” (Vox 2022) which they’re trying to turn around via moving above 10% of funding directly to local organizations.
How many ex-post project closure evaluations have been done? .002% of all projects. The 0.002% statistic looks at just public foreign development aid from 1960 (not even counting private funding such as foundations or gifts to organizations, which isn’t tracked in any publicly available database). Calculating aggregated OECD aid statistics (excluding private because it’s only recent data) over 62 years $5.6 trillion by 2022 (thanks to Rebecca Regan-Sachs for the updated #s).
I then estimated 3.000 actual ex-posts which comes from 2,500 JICA projects plus almost 500 other projects that I have either found looking through databases all across the spectrum from governments and multilaterals (almost 100 in our catalysts, and am assuming there must be 400 others done in the 1980s-2000 like USAID and the World Bank).
Without a huge research team it is improssible to aggregate data on the total number of projects by all donors. So I extrapolated from project activity disbursements of one year (2022) for Mali on the www.foreignassistance.gov page. In my 35 years of experience, Mali, where I did my doctoral research, typifies he average USAID aid recipient. They had 382 projects going in 2022. I rounded up to 400 projects x 70 years (since 1960 when OECD data began) x 100 countries by just one donor (of the 150 possible recipient countries, to be conservative). This comes to 2.8 million projects. So if we take 39 OECD countries as donors (given most have far less to give than US), in total 109 million publicly funded aid projects disbursed $5.6 trillion since 1960. While final evaluations are industry standard, only .002% is the estimated number of ex-post evaluations of projects the were evaluated with data from local participants and partners of the 109 million projects .
This became Valuing Voices focus, and we created an open-access database for learning, and conducted our own My team and I identified 92 ex-posts that returned to ask locals what lasted, what didn’t, why, and what emerged from their own efforts. We also created evaluability checklists and created a new evaluation, Sustained and Emerging Impacts Evaluation that included examining not just what donors put in place to last, but also what emerged outcomes from local efforts to sustain results with more limited resources, partnerships, capacities and local ownership/motivation. These four drivers were found by Rogers and Coates for USAID’s food security exit study in 2015). We have done 15 ex-posts for 9 clients since 2006 and shared Adaptation Fund ex-post training materials in 2023.
Yet the public assumes we know our development is sustainable. 2015’s ‘Sustainable Development Goals‘ focused aid on 17 themes, which was to generate $12 trillion more in annual spending on SDG sectors than the;$21 trillion already being invested each year. Nonetheless, a recent UN report states that there is now a $4 trillion annual financing gap to achieve the SDGs. All this funding goes to projects that are currently implemented, not to evaluate what had been sustained from past projects that already closed. Such learning from what succeeded or failed, or what emerged from local efforts to keep activities and results going is pivotal to improving current and future programming is almost wholly missing from the dialogue; I know, I asked multiple SDG evaluation experts.
Why do we return to learn so rarely? There are many reasons, the most prosaic among them being administrative.
- When aid funds are spent over 2-10 years, projects are closed, evaluated at the end, ‘handed over’ to national governments, and no additional funding exists to return ‘ex-post’ closure to learn.
- Next is the push to continue to improve lives through implementation which means low rates of overhead allocated to M&E and learning during, much less after closure.
- Another is the assumption that ‘old’ projects differ so much from new ones, but there are few differences. After all there are only so many ways to grow food, feed the malnourished, educate children; evaluating ‘old’ projects can teach ‘new’ projects.
- A last major one, from Valuing Voices’ research of 12 years may be the largest: Fear of admitting failure. Please read Valuing Voices’ 2016 blog highlighted many Lessons about Funding, Assumptions and Fears (Part 3). One US aid lobbyist told me in 2017 that I must not share this lack of learning about sustained impacts because it could imperil US aid funding; I told her I had to tell people because lives were at stake.
- Overall, there is much to learn; most ex-post evaluations show mixed results. None show 100% sustainability and while most show 30-60% sustainability, none are 0% sustained either. If we don’t learn to replicate what worked and cease what didn’t now, then future programming will be as flawed and successes, especially brilliant emerging locally designed ex-post outcomes such as Niger’s local funding of redesign of health incentives will remain hidden.
Occasionally donors invest in sets of ex-post learning evaluations such as USAID’s ‘global waters’ seven water/ sanitation evaluations linked to the E3 Bureau taking sustainability as a strategic goal. Yet the overall findings from USAID’s own staff of these ex-posts Drivers of WASH study were chilling. While 25 million gained access to drinking water and 18 million to basic sanitation, ‘they have largely not endured.’ But the good news in such research is that the donor learned that infrastructure fails when spare parts are not accessible and maintenance not funded or performed, which can be planned for and addressed during implementation by investing in resources and partnerships. They learned that relying on volunteers is unreliable and management needs to be bolstered, which can lead to some implementation funding to be focused on capacities and local ownership. We can plan better for sustainability by learning from ex-post and exit studies (see Valuing Voices’ checklists in this 2023 article on Fostering Values-Driven Sustainability).
And since 2019, three climate funds, the Adaptation Fund, the Global Environmental Facility, and the Climate Investment Funds have turned to ex-post evaluations to look at sustainability and longer-term resilience and even transformation, given environmental shocks may take years to affect the project sites. The Adaptation Fund has done four ex-posts, with more to come in 2024/25, and the CIF is beginning now. The GEF has done a Post-Completion Assessment Pilot for the Yellow Sea Region . Hopeful!
by Jindra Cekan | Sep 24, 2021 | Aid effectiveness, Bill and Melinda Gates Foundation, Donors, emerging outcomes, Evaluation, ex-post evaluation, Impact, Lutheran World Relief, Nepal, Niger, Sustainability, Sustainable development
Aid providers: More puzzle pieces, including unexpected outcomes; ours is not the whole picture
When we did our first ex-post evaluation/ delayed final evaluation in 2006 in Niger for Lutheran World Relief (LWR) funded by the Bill & Melinda Gates Foundation (pg75 on), we found all sorts of unexpected/ unintended outcomes and impacts that far outweighed the original aid’s expectations. The project measured success by the livelihoods rebuilding post-drought ravaged sheep herds and water points for them. Instead, while the LWR aid was beautifully based on ‘habbanaye,’ a pastoralist practice of lending or giving small-stock offspring to poorer family members (and was expanded to passing on animals to poorer community members), and results showed a majority of the poor benefitted, our respondents showed it was far more nuanced. We aid providers (and our expectations) are only a part of any ‘impact’, which needs to be defined by the communities themselves:
- While many families benefitted from the sheep, enabling some young boys to shepherd several at a time, it turns out the poorest chosen by the communities were not necessarily ones who lost small livestock during the drought but were, in fact, the ultra-poor who had never had them. Therefore, proof of successful ‘restocking’ post-drought left these poor who were helped the most, out, as they were… unexpected;
- Holding onto the donated sheep was not as important an indicator for some: one woman told us that selling her aid-received sheep to buy her daughter dowry to marry a wealthier husband was a far better investment for their financial future than the sheep would have been. Our main measure of success was not nuanced enough;
- The provision of water through well-rehabilitation and building in the five villages was a vital resource. Women reported they saved 8 hours every two days by having potable water in their villages. Before then, they spent three hours walking each way to the far-off well and waited two hours to fetch 50l water, which they head-carried back. With the well in place, they generated household income through weaving mats, cooking food for sale, etc., which amounted to as much as 20% of increased household income – a boon! Also, having both time and water access enabled them to bathe themselves and their children, make their husbands lunch, and make their mothers-in-law tea, which led to far more household harmony. No ‘impact’ measures outside of livestock and water were included or could be added;
- Finally, the resulting show-stopper: in the last village where we interviewed women participants, they said that the groups of fellow recipients were a boon for community solidarity across ethnic groups. In their meetings, thanks to the sheep, water, and collective moral support, they said the conversations turned from conflict to collaboration, and best of all, women reported, “our husbands don’t beat us anymore.” Peace among ethnic groups, within and between households was completely unexpected.
Such a highly unexpected outcome would fall under #2 and #4 of the Netherlands study below. Unfortunately, the Foundation seemed less interested in these unexpected but stellar results. Yet at the same time I have empathy for their position, as so many of us in global development want to help solve problems, and demand proof we have.…. so we can leave and help others, equally deserving. Taking complexity into account, seeing lives in a wider context where our aid can be helping differently or even harming makes garnering more aid hard.
As the brilliant Time to Listen series by CDA showed, aid intervenes in people’s lives in complex ways and we need to listen to our participants and partners who always share more complex views than our reports can honor..A few of hundreds of quotes of 6000 interviews, this from the Solomon Islands:
Some appreciate the aid as it is given:
“People in my village are very grateful for the road because now with trucks coming into our village, the women can now take their vegetables to the market. Before, the tomatoes just rotted in the gardens. Tomatoes go bad quickly and despite our attempts in the past to take them to the market to sell, we always lost.” Woman from East Malaita
But for others there are great caveats:
“Donors should send their officers to Solomon Islands to implement activities in urban and rural areas. This will help them understand the difficulties we often face with people, environment, culture, geography, etc. ‘no expectem evri ting bae stret’ [Don’t expect everything to go right].” Man, Honiara
“They have their own charters, sometimes we might want to go another way but they don’t want to touch that. So sometimes there is some conflict there; some projects are not really what we would like to address – because the donors only want to do one component, and not another, because it is sensitive, or because they want quick results and to get out.” Government official, Honiara
“What changes have I noticed since independence? Whatever development you see here is due to individual struggles. No single aid program is sustainable. NGOs are created by donors and are comfortable with who they know. NGOs eat up the bulk of help intended for the communities. NGOs become international employers. They do their own thing in our province. Most projects have no impact. I want to say stop all aid except for education and health. If international assistance concentrates on quality education and health, the educated and healthy people will take care of themselves.” Government official, Auki, Malaita
“The most important impacts of aid people do not think about – they are not listed, not planned, they are remote, but these are the longest lasting. Often they are the opposite of the stated objectives. So remote, unintended, unexpected impacts are very often more important and more lasting and more dramatic than the short term intended, measured ones.” Aid consultant, Honiara
How widespread is our myopic focus on our intended results? A recent Netherlands Foreign Aid IOB study found unintended effects were an evaluator’s blindspot as across 664 evaluations over 20 years, “The ‘text miners’ found that only 1 in 6 IOB evaluation documents pay attention to unintended effects.” This dearth of attention to all the other things happening in projects led to 10 micro, macro, meso, and multiple level effects, from negative price effects such as food aid on local food producers or nationalist backlash to Afghan projects to positive effects (they found 40% of projects had this) such as a harbor built happening to expand beach tourism as well.
But if we don’t look for such effects, we don’t know the true impact of our aid programming. We also don’t honor the breadth of people’s lives rather than just as narrow ‘aid beneficiaries’ (ugh), not even honoring them with the terms’ participants’ much less ‘partners’ in their own development).
In our ex-post work, we find a wide array of ways ownership and implementation of activities is done after donors leave and without additional or with different resources, capacities, and partnerships. Taking emerging outcomes and impacts examples from a different Niger project, and one from Nepal:
1. Partnerships & Ownership: Half of the members of the all-women Village Banks reported helping one another deal with domestic disputes and violence. (Pact/Nepal)
2. Capacities: Trained local women charged rates to sell course materials onward (PACT/Nepal)
3. Ownership: Participants valued clinic-based birthing and sustained it by introducing locally-created social punishments and incentives (CRS/Niger)
4. Resources: New Ministry funding reallocated to sustain [health] investments, and private traders generated large crop purchases and contracts (CRS/Niger)
The assets and capacities we bring to help people and their country systems help only a sliver of their lives, and often in unexpected ways that sadly we aid donors and implementers don’t seem interested in. There are other puzzle pieces to add…
Let us not forget, as a Sustainable Development Goals Evaluation colleague said in 2017 on a call:
I am sure you, my dear colleagues, have reams of similar findings from your fieldwork. Please share yours!
by Jindra Cekan | Oct 8, 2016 | Corporate Social Responsibility, CSR, Donors, Exit strategies, Social Investment, South Africa, Sustainability, Sustainable development
I am delighted to repost the blog on Responsible Donor Exit from Tshikululu, a Social Investment advisory firm in South Africa that I met at the European Evaluation Society conference last week in Holland [1]. The short report outlines different choices of Phase down, Phase out, or Phase over for Corporate Social Responsibility (CSR). As we in foreign aid evaluation have noted, donors should have set criteria for engaging with grantees to facilitate the transition and exit from programs, including exit plans and designing exit in from the beginning. What this report adds is open communication and joint agreements on timelines, exit grants and post-exit ‘scans’ that may foster further partnerships. Especially pleased to see them recommending “programme beneficiaries should therefore be empowered to direct the development processes that affect them.” Enjoy…
Towards responsible donor exiting strategies and practices
10 May 2016 | Silvester Hwenha |
Social investment has evolved as the result of a number of factors, including a growing interest by high net worth individuals and institutional investors in tackling social issues at the local, national or global level. Social investors have also become increasingly relevant in many countries as a result of mounting social challenges amid declining public funds to provide social services. The rationale for social investment is based on the realisation that social or environmental factors can impact a company’s bottom line and therefore are important factors in business. Besides, it has long been acknowledged by civic society and business that government alone cannot confront and solve all of society’s problems.
Social investors typically channel their funds through non-profit entities including non-governmental organisations (NGOs) and community based organisations (CBOs) to deliver social and environmental programmes in communities where such programmes are required. However, while social challenges require long term interventions to address, social investments often support programmes in short funding cycles. In many instances social investment funds are redirected to other social challenges thus necessitating exiting of programmes.
Exiting programmes is usually a highly sensitive and difficult process for donors, grantees and beneficiaries. For most donors, the reasons for exiting programmes include changing priorities and/or leadership, dwindling resources and the potential threat on programmes by the emergence of political instability. Despite having legitimate cause to exit programmes, donor agencies, foundations, trusts and corporate donors often do so with little advance notice, communication and consultation with programme partners.
Source:
[1] Hwenha, S. (2016, May 10). Towards responsible donor exiting strategies and practices. https://web.archive.org/web/20161002091204/http://www.tshikululu.org.za/insights-opinions/entry/towards-responsible-donor-exiting-strategies-and-practices
by valuingvoicesjin | Feb 24, 2015 | Accountability, Aid effectiveness, Beneficiaries, Charity Navigator, DFID, Donors, Evaluation, Feedback loops, Fund for Shared Insight, impact evaluation, Irene Guijt, Keystone, Participants, People First, post-project evaluation, Sustainable development, World Bank
Listening better… for more sustainable impact
Are we listening better? Maybe. As Irene Gujit states on Better Evaluation, Keystone’s work on ‘constituent voice’ enables a "shift [in] power dynamics and make organizations more accountable to primary constituents”. For example, "organisations can compare with peers to trigger discussions on what matters to those in need… in (re)defining success and ‘closing the loop’ with a response to feedback [on the project], feedback mechanisms can go well beyond upward accountability."
There are impressive new toolkits available to elicit and hear participant voice about perceived outcomes and impacts, such as People First Impact Method and NGO IDEAS' Monitoring Self-Effectiveness. As People First states, "Across the aid sector, the voices of ordinary people are mostly not being heard. Compelling evidence shows how the aid structure unwittingly sidelines the people whom we aim to serve. Important decisions are frequently made from afar and often based on limited or inaccurate assumptions. As a result, precious funds are not always spent in line with real priorities, or in ways that should help people build their own confidence and abilities…. As a sector, we urgently need to work differently." These are results of 40 year old participatory/Rapid Rural Appraisal distilled and shared by IDS/UK's Robert Chambers which I've used for 25 years, including lately for self-sustainability evaluation.
In addition to qualitative, participatory tools, the application of quantitative evaluative tools have a ways to grow to be terrific at listening and learning. Keystone did interesting work on impact evaluation (lately associated with Random Control Trials comparing existing projects and comparable non-participating sites to prove impact). Their study found that not only "no one engaged through the research for this note is particularly happy with the current state of the art…. There is a strong appetite to improve the delivery of evaluative activities in general and impact evaluation in particular … Setting expectations by engaging and communicating early and often with stakeholders and audiences for the evaluation is critical, as is timing." So many of us believe that evaluation cannot be an afterthought, but monitoring and evaluation needs to be integrated into project design, with feedback loops informing implementation.
Yet this otherwise excellent article made one point that is common, yet like Alice looking through the looking glass backwards. For they write feedback is "to inform intended beneficiaries and communities (downward accountability) about whether or not, and in what ways, a program is benefiting the community". Yet it is the other way around! Only communities have the capacity to tell us how well they feel we are helping them!
Thankfully, we are increasingly willing to listen and learn about aid effectiveness. Some major actors shaping funding decisions have already thrown down the feedback gauntlet:
* As our 2013 blog asked for, Charity Navigator is now applying its new “Results Reporting” rating criteria, which include six data points regarding charities feedback practices. The new ratings will be factored into Charity Navigator star ratings from 2016.
* Heavyweight World Bank president Jim Kim has decreed that the Bank will require robust feedback from beneficiaries on all projects for which there is an identifiable beneficiary.
* The Hewlett, Ford, Packard, Rita Allen, Kellogg, JPB and LiquidNet for Good Foundations have recently come together to create the Fund for Shared Insight to catalyze a new feedback culture within the philanthropy sector.
* This February, a new report on UK's international development agency, DFID recommended a new direction to their aid: "The development discourse has generally focused on convincing donors to boost their aid spending, when the conversation should instead be on “how aid works, how it can support development, how change happens in countries, and all of the different responses that need to come together to support that change…. One important change will be for professionals to deliver more adaptive programming and work in more flexible and entrepreneurial ways… emphasized the need for development delivery to be led by local people. Commenting on ODI’s research, [DFID} said successful development examples showed “people solving problems for themselves rather than coming in and trying to manage that process externally through an aid program.”
Hallelujah! What aid effectiveness great listening are you seeing?
by Jindra Cekan | Feb 13, 2015 | Accountability, Aid effectiveness, Bilateral organizations, Czech, Donor Driven Development, Donors, Evidence-based policy, ex-post evaluation, International aid, NGO, post-project evaluation, Republic of Georgia, self-sustainability, Sustainable development
Czech it out! Great evaluation happening in the Czech Republic
One of the delights of living in another country is the surprises one encounters. For me, coming back to our second 'home', it was an evaluative surprise. For by connecting to the Czech Foreign Ministry's Evaluation team, I found evidence of learning from meta-evaluation, doing ex-post evaluation, conscientious tracking of project cost-effectiveness and an openness to self-sustainability research funding and using national evaluators to lead them.
Czech Foreign aid is widespread- "Through development cooperation, the Czech Republic helps to eradicate poverty in less developed parts of the world by means of sustainable socio-economic development. It also contributes to global security and stability, conflict prevention, the promotion of democracy, human rights and fundamental freedoms, and the rule of law". Development assistance is done by several entities, the main two under the Ministry of Foreign Affairs, ORS (Development Cooperation and Humanitarian Aid) and its subsidiary CRA (Czech Development Agency).
The Ministry of Foreign Affairs oversees some fascinating evaluation work. After attending several partner-donor meetings and presentations of a meta-evaluation, an ex-post from an array of projects in Georgia and a discussion of findings across all evaluations in 2014, I am impressed. Why? Because not only are they willing to learn from both successes and failures, openly discussing challenges in learning between grants and contracts, but also because they are tackling programming in 10 countries (e.g. Afghanistan, Bosnia and Herzegovina, Ethiopia, Georgia) with a mere 17 people and a budget of $35 million for 2015.
What are some of the things they are learning?
1. They commissioned a meta-evaluation looking at 20 projects from 2012-2013. What worked well was well described and documented evaluations that were also cost-effective (evaluations were 4% of total costs) and tried to offer constructive solutions to things that did not work well in projects. While some methodologies needed to be better, and reports were hard to access, a major finding was what needs to be improved is inclusion of local recipients in stakeholder analyses, soliciting their views on what the evaluations should focus and on how the projects affected them. Further, during discussions we highlighted the need for an evaluation of outcomes and impacts, not just how evaluations quality was but also which organizations had the best results and why.
2. They commissioned an ex-post evaluation across eight organizations' in the Republic of Georgia (5 Czech, 3 Georgian), of one-year projects with 131 separate activities in civic engagement, media and youth between 2008-2012. The evaluation looked at the short-term effectiveness and longer-term sustainability of activities in the Republic of Georgia. Key findings included good relevance of aid offered, high cost-efficiency, low effectiveness for Georgian decision-making, primarily individual (rather than systemic) sustainability, though some good impact.
Key recommendations from this evaluation-, which Valuing Voices thinks, are universal included:
LEARN BETTER TOGETHER
a. Implement min. 3-year projects, whereby focus in a selected region (or a few regions) on a selected local priority topic, ensure in-depth needs analysis, multi-stakeholder cooperation [including participants], sustainable mechanisms, ongoing local support and enough flexibility as per external factors.
b. Allocate budget for burning human rights issues and for enhancing planning, monitoring, evaluation and learning capacities of Civil Society Organizations.
and
SHARE FINDINGS MORE
c. Coordinate activities with other implementers and donors in the target area and if possible (taking into account the political situation) also with local state institutions and potential implementers
d. Implement multi-stakeholder initiatives in a specific area (health, environment, social inclusion, minorities) with an advocacy component, sharing of results / lessons learnt and a media component
3. Among annual recommendations from Evaluation discussions throughout the year emerged this surprising one on cost-effectiveness. A detailed financial budget is now standard, and expenses for project activities among a majority of (grant-funded) projects and the Czech Development Agency are now required. This enables cost-effectiveness comparisons at least across grants (albeing not across for-profit contracts). In my experience this is unparalleled! (Let me know if other countries do this please!)
Overall, the fact that the Czech Foreign Ministry and implementing partners are willing to look at themselves critically and transparently improve accountability to its ultimate recipients and taxpayers makes me shout Hurrah from all of Prague’s 100 spires! Here is one of them, taken from a Ministry’s window.
