by Jindra Cekan | Dec 17, 2021 | Adaptation Fund, climate change, Ecuador, ex post evaluation, ex-post evaluation, Food security, Impact, infrastructure, Monitoring and Evaluation, or, Resilience, Samoa, SEIE, Sustainable development, sustained outcomes, UNDP, WFP
Follow our path on Ex-Post Sustainability and Resilience Seasonal Joy
Rarely do we get to teach, innovate, learn, and expand a hidden corner of one’s field. This is what is I am experiencing, and the Fund is sharing. Here is the Adaptation Fund’s ex-post project evaluation of sustainability and resilience path and progress in 2021.
I am quoting liberally and highlighting our work from the Adaptation Fund’s website where their commitment to learning from what lasts is clear. “Ex post evaluations are a key element of the AF-TERG FY21-FY23 strategy and work programme, originating from the request of the Adaptation Fund Board to develop post-implementation learning for Fund projects and programmes and provide accountability of results financed by the Fund. They intend to evaluate aspects of both sustainability of outcomes and climate resilience, and over time feed into ex-post-evaluation-informed adjustments within the Fund’s Monitoring Evaluation and Learning (MEL) processes.”
How are we defining sustainability’s path to evaluate it? Here is a flowchart from our training:
There are four phases from 0 to 3:
Phase 0 Foundational Review: Not only was this work preceded by months of background research on both evaluability of their young portfolio (e.g., under 20 of the 100 projects funded were closed at least three years, a selection criteria we had) and secondary research on evidence of ex-post sustainability evaluation in climate change/ resilience across the Adaptation Fund’s sectors.
Phase 1 Framework and Pilots Shortlist: Our Phase 1 report from mid-2021 provided an overview of the first stage of ex-post evaluations, outlining methods and identifying a list of potential projects for ex-post evaluation pilots from the Fund’s 17 completed, evaluated projects. The framework presented in the report introduced possible methods to evaluate the sustainability of project outcomes, considering the characteristics, strengths, and weaknesses of the Fund portfolio. It also presents an analysis tool to assess climate resilience, bearing in mind that this area is pivotal to climate change adaptation yet has rarely been measured.
The Phase one report on ex post project sustainability evaluation
Vetting and pilot selection, revised design for evaluating sustained outcomes related to resilience to climate change. Key aspects are: 1) Timing (3-5 years since closure or projects at least 4 years long within the last 5 years and seasonality matches the final evaluation) and 2) Good quality of implementation and M&E with measurable outputs and outcomes traceable to impact(s) and 3) Safety to do fieldwork re: Covid, civil peace, etc.
We (my so-clever colleagues Meg Spearman and Dennis Bours) introduced a new resilience analysis tool that includes consideration of the climate disturbances, the human and natural systems (and their nexus) affected by and affecting project outcomes. This includes five characteristics of resilience in the outcomes (presence of feedback loops, at scale, plus being diverse, dynamic, and redundant) and means/actions to support outcomes. Resilience can be identified via a clear summary of the structures (S) and functions (F) that typify Resistance, Resilience and Transformation showing where a project is and is moving towards. It is a typology of resistance-resilience-transformation (RRT) onto which the overall project can be mapped based on how actions are designed to maintain or change existing structures and functions. That was integrated into the Adaptation Fund resilience evaluation approach.
Phase 2 Methods Testing and Ex-post Field-testing: Training of national evaluators and piloting two ex-post evaluations per year includes selecting among these methods to evaluate sustainability ex-post plus the RRT and resilience measures above. In the first ex-post in Samoa’s “Enhancing Resilience of Samoa’s Coastal Communities to Climate Change” (UNDP) happening December 21, it is through qualitative evaluation of wall-infrastructure. The second, Ecuador’s “Enhancing resilience of communities to the adverse effects of climate change on food security, in Pichincha Province and the Jubones River basin “(WFP) has training completed and fieldwork should be from January 22, likely be of food security assets and methods TBD.
Phase 3 Evaluations continue, with MEL Capacity Building: Two more years of ex-post pilot evaluations (2 per year) with lesson informing integration into the MEL of the Adaptation Fund. We are already finding out lessons of rigor, of knowledge management, of unexpected benefits of returning years after closure, including indications of sustainability and resilience of the assets, with much more learning to come.
Innovations include “the relative novelty of climate change adaptation portfolios and the limited body of work on ex post evaluation for adaptation, it presents possible methods that will be piloted in field-tested ex post evaluations in fiscal year 2022 (FY22).” This includes piloting shockingly rare evaluation of oft-promised resilience. In the update to AF’s Board three months ago, it transparently outlined shortlisting of five completed projects as potential candidates for the pilots, of which two projects were selected for ex post evaluations. It outlined our process of co-creating the evaluation with national partners to prioritize their learning needs while building national capacity to assess sustainability and resilience of project outcomes in the field onward.
Also, training materials for ex post pilots are being shared to foster country and industry learning, focusing on evaluating projects at ex-post and emerging sustainability and resilience, as well as presenting and adapting methods to country and project realities.
The training material for ex-post pilots
The training had three sessions (which could not have happened without colleague Caroline’s expertise):
- Part A: Understanding ex-post & resilience evaluations. Introduce and understand ex-post evaluations of sustainability and resilience, especially in the field of climate change adaptation
- Part B: Discussing country-specific outcome priorities and co-creating learning with stakeholders. Discuss the project and its data more in-depth to understand and select what outcome(s) will be evaluated at ex-post
- Part C: Developing country-specific methods and approaches. Discuss range of methods with the national evaluator and M&E experts to best evaluate the selected outcome(s) and impact(s)
Overall progress can be seen in the document updating the AF’s board: A progress update on ex post evaluations (AFB/EFC.28/Inf.4)
So in 2018 I Wished for Sustained Outcomes to be explored and in 2019 I Wished again for more Ex-post Evaluation than Needles in Haystacks. In 2021 some of these wishes are becoming fulfilled! Seasonal Happiness for me is learning about resilience to climate change, diminished vulnerability and searching for proof of sustainability and emerging outcomes and impact(s) and I am grateful to the Adaptation Fund for its commitment to sustainability.
Accompany us on this path, cheer us on, and do your own ex-post sustainability and resilience evaluations! Happy holidays from the Czech Republic!
by Jindra Cekan | Oct 28, 2021 | Accountability, Adaptation Fund, Aid effectiveness, climate change, environment, foreign aid, Niger, Sustainability, Sustainable development
5 Ways to Foster Sustainability and Resilience to Climate Change
by Omar Abdou, M.A. and Jindra Cekan/ova, Ph.D
In the global climate change graphic, the blue are countries that contribute very few emissions to #climatechange. They are also those who suffer the most from climate change effects and who have most #globalaid projects which are most at risk of no longer being #sustained:
https://globalgovernanceforum.org/global-issues/climate-change/
Omar Abdou, a food security and M&E expert from Niger, lives a vulnerability to climate change there is indisputable. Sahelien production systems suffer the full brunt of the negative effects. Indeed, the essential rural activities, namely agriculture, and livestock which occupy at least 80% of the population, are becoming more and more uncertain as they rely on two pillars: rainfall and the exploitation of natural resources (soil, vegetation, other water sources). But, in recent decades, we have witnessed a drastic reduction in rainfall. In pastoral zones, this decrease is reflected in the degradation of the vegetation cover and the deficits in livestock-vital fodder. For instance, the fodder assessment from 2000 to 2020 for the Tahoua Region has been in deficit for 15 years. More and more recurrent and frequent crises seriously affect the main productive capital of households, which is the herd.
https://intraacpgccaplus.org/story/niger-centre-studies-climate-smart-agriculture-and-climate-finance/
Emergency sales of increasingly emaciated animals at a low price are growing to pay for consumer goods and to buy fodder to feed a few animals. These are kept for breeding after the crisis, but what used to be seasonal crisis (dry-season) has become chronic.
In agro-pastoral zones, agricultural production in the rainy season, which has changed its duration and strength, becomes uncertain to feed the household members. The harvest is often exhausted at most by the 6th month of the year for many households. Thus, there is no more food surplus to supply the area’s food needs where cereals are not grown. Suddenly, cereals become expensive and inaccessible for many families. Their coping strategies are diverse but not always effective. In pastoral settings, parents are forced to take their children out of school to go further south in search of food and pasture. Sometimes more sedentary agro-pastoralist farmers pledge or sell their land to pay for food. Those left behind who have nothing to sell, are forced to cut trees and sell the wood for charcoal, further degrading the environment. Unfortunately, Niger’s neighbors are now mostly at war, so these young men run the risk of being recruited by armed groups or are killed. In these conditions, there is serious instability where the means of production (natural and human) are sold for survival and overexploited natural resources decrease. Increasingly, it is practically impossible to talk about sustainability. It is, therefore, necessary to think of consistent and innovative projects to enable households to produce sustainably while preserving the ecosystem for future generations. Outlets such as migration of the able-bodied workers to neighboring countries to provide labor in the lean season work now but are unsustainable.
Jindra notes that 31 Oct the UN Climate Change Conference (COP26) begins. Climate science documents life-threatening effects such as Omar and billions of others are already experiencing around the ‘developing’ world. The IPCC Working Group report shows that “the world will probably reach or exceed 1.5 degrees C (2.7 degrees F) of warming within just the next two decades. Whether we limit warming to this level and prevent the most severe climate impacts depends on actions taken this decade. Only with ambitious emissions cuts can the world keep global temperature rise to 1.5 degrees C, the limit scientists say is necessary for preventing the worst climate impacts. Under a high-emissions scenario, the IPCC finds the world may warm by 4.4 degrees C by 2100 — with catastrophic results.” Erratic rainfall, greater heatwaves will affect 70% of all farmers and herders who rely on rain for their production and a new report on Adaptation to climate across West African shows aridity increasing, which must inform global development.
What can we do:
1- LISTEN AND ACT GLOBALLY: There are excellent resources to track demands that the Global North divest and stop subsidizing fossil fuels. For instance, “In 2017/18, the G7 collectively spent more on fossil fuel finance than on climate finance, according to analysis by Oil Change International. They allocated nearly $40bn that year.” Hypocrisy abounds as this Climate Change News article goes on to note that “Since the Paris Agreement was signed, G20 countries’ export credit agencies have provided 14 times as much support for fossil fuels as clean energy “to $64 billion a year, and investors still prioritize profit over planet & people.
2- LISTEN AND PRIORITIZE GLOBAL SOUTH VOICES: While funding demands for over $100billion to LDCs is a demand of COP26, the need to support local research and activist organizations who show measured results is mostly missing (more from Sweden here). Other resources include WRI’s Allied for Climate Transformation by 2025 podcast series among the most Climate Change vulnerable nations. “ACT2025 consortium to ensure voices from countries most exposed to climate change are heard, empowered, mobilized, and adequately supported in international climate negotiations.“ The series includes how countries such as Niger adapt to climate change. If nothing else, consider the tragedy of the loss of $1.5 billion in aid invested in Niger, not to mention the ensuing suffering global Northern consumption is unleashing on the global South.
3- DESIGN CLIMATE SMARTLY: Global Development needs to do much more climate-smart programming as 50 years of investments in foreign aid to ‘developing’ countries has already started to be threatened. What can be done? From sequestering the carbon content of soil via farming to fostering climate-smart agriculture, which, according to the UN’s FAO means “agriculture that sustainably increases productivity, enhances resilience (adaptation), reduces/removes GHGs (mitigation) where possible, and enhances achievement of national food security and development goals”. Importantly this involves productivity plus adaptation and mitigation against changes wrought by climate change.
4- MEASURE SUSTAINABILITY: Exciting measurements are beginning! Scotland is valuing its nature, and there are calls for The Rights Of Nature In Evaluation Of Environmental Sustainability. Jindra is consulting to a wonderful team at the Adaptation Fund. We are evaluating what could be sustained, and what withstood climatic changes is ex-post project sustainability and resilience evaluation. The Adaptation Fund on evaluating projects’ longer-term (3-5 year ex-post project closure) sustainability and, importantly their resilience to climate change. We are piloting these rare kinds of evaluations in six countries, starting with Samoa and Ecuador in 2021. As the Fund has funded over 100 projects in 100 countries, with almost 30 million participants, we have a rich array to choose from. Such a learning opportunity about what lasts and has withstood disturbances is vital. As the Fund notes: “Climate change is predicted to greatly affect the poorest people in the world, who are often hardest hit by weather catastrophes, desertification, and rising sea levels, but who have contributed the least to the problem of global warming. In some parts of the world, climate change has already contributed to worsening food security, reduced the predictable availability of freshwater, and exacerbated the spread of disease and other threats to human health. Helping the most vulnerable countries and communities is an increasing challenge and imperative for the international community, especially because climate adaptation requires significant resources beyond what is already needed to achieve international development objectives.”
5- TAKE PERSONAL RESPONSIBILITY: For 8 years, Valuing Voices has pushed back to evaluate the ‘Sustainable Development” we have promised for 50 years. We must be accountable primarily to our participants and partners, rather than donors, and only by listening to these country nationals, we can learn what was sustained not, why or why not, and what emerged from their efforts. In 2019, we wrote all of us are accountable to improve the SDGs. It is wonderful to have climate change leading organizations invest in ex-post evaluation that adds resilience, prioritizing organizational learning and accountability. We all must increase our awareness that curb our consumption, for the sake of Niger, small island states, and many other parts of our planet can no longer stand our business-as-usual. Limiting climate change begins with us, from changing our food consumption to consumer choices to political advocacy to seeing the Earth as indivisible from us and us from the Earth. Let us start to invest in global sustainability today.
Do you see any bright spots on the horizon? Do share!
by Jindra Cekan | Sep 24, 2021 | Aid effectiveness, Bill and Melinda Gates Foundation, Donors, emerging outcomes, Evaluation, ex-post evaluation, Impact, Lutheran World Relief, Nepal, Niger, Sustainability, Sustainable development
Aid providers: More puzzle pieces, including unexpected outcomes; ours is not the whole picture
When we did our first ex-post evaluation/ delayed final evaluation in 2006 in Niger for Lutheran World Relief (LWR) funded by the Bill & Melinda Gates Foundation (pg75 on), we found all sorts of unexpected/ unintended outcomes and impacts that far outweighed the original aid’s expectations. The project measured success by the livelihoods rebuilding post-drought ravaged sheep herds and water points for them. Instead, while the LWR aid was beautifully based on ‘habbanaye,’ a pastoralist practice of lending or giving small-stock offspring to poorer family members (and was expanded to passing on animals to poorer community members), and results showed a majority of the poor benefitted, our respondents showed it was far more nuanced. We aid providers (and our expectations) are only a part of any ‘impact’, which needs to be defined by the communities themselves:
- While many families benefitted from the sheep, enabling some young boys to shepherd several at a time, it turns out the poorest chosen by the communities were not necessarily ones who lost small livestock during the drought but were, in fact, the ultra-poor who had never had them. Therefore, proof of successful ‘restocking’ post-drought left these poor who were helped the most, out, as they were… unexpected;
- Holding onto the donated sheep was not as important an indicator for some: one woman told us that selling her aid-received sheep to buy her daughter dowry to marry a wealthier husband was a far better investment for their financial future than the sheep would have been. Our main measure of success was not nuanced enough;
- The provision of water through well-rehabilitation and building in the five villages was a vital resource. Women reported they saved 8 hours every two days by having potable water in their villages. Before then, they spent three hours walking each way to the far-off well and waited two hours to fetch 50l water, which they head-carried back. With the well in place, they generated household income through weaving mats, cooking food for sale, etc., which amounted to as much as 20% of increased household income – a boon! Also, having both time and water access enabled them to bathe themselves and their children, make their husbands lunch, and make their mothers-in-law tea, which led to far more household harmony. No ‘impact’ measures outside of livestock and water were included or could be added;
- Finally, the resulting show-stopper: in the last village where we interviewed women participants, they said that the groups of fellow recipients were a boon for community solidarity across ethnic groups. In their meetings, thanks to the sheep, water, and collective moral support, they said the conversations turned from conflict to collaboration, and best of all, women reported, “our husbands don’t beat us anymore.” Peace among ethnic groups, within and between households was completely unexpected.
Such a highly unexpected outcome would fall under #2 and #4 of the Netherlands study below. Unfortunately, the Foundation seemed less interested in these unexpected but stellar results. Yet at the same time I have empathy for their position, as so many of us in global development want to help solve problems, and demand proof we have.…. so we can leave and help others, equally deserving. Taking complexity into account, seeing lives in a wider context where our aid can be helping differently or even harming makes garnering more aid hard.
As the brilliant Time to Listen series by CDA showed, aid intervenes in people’s lives in complex ways and we need to listen to our participants and partners who always share more complex views than our reports can honor..A few of hundreds of quotes of 6000 interviews, this from the Solomon Islands:
Some appreciate the aid as it is given:
“People in my village are very grateful for the road because now with trucks coming into our village, the women can now take their vegetables to the market. Before, the tomatoes just rotted in the gardens. Tomatoes go bad quickly and despite our attempts in the past to take them to the market to sell, we always lost.” Woman from East Malaita
But for others there are great caveats:
“Donors should send their officers to Solomon Islands to implement activities in urban and rural areas. This will help them understand the difficulties we often face with people, environment, culture, geography, etc. ‘no expectem evri ting bae stret’ [Don’t expect everything to go right].” Man, Honiara
“They have their own charters, sometimes we might want to go another way but they don’t want to touch that. So sometimes there is some conflict there; some projects are not really what we would like to address – because the donors only want to do one component, and not another, because it is sensitive, or because they want quick results and to get out.” Government official, Honiara
“What changes have I noticed since independence? Whatever development you see here is due to individual struggles. No single aid program is sustainable. NGOs are created by donors and are comfortable with who they know. NGOs eat up the bulk of help intended for the communities. NGOs become international employers. They do their own thing in our province. Most projects have no impact. I want to say stop all aid except for education and health. If international assistance concentrates on quality education and health, the educated and healthy people will take care of themselves.” Government official, Auki, Malaita
“The most important impacts of aid people do not think about – they are not listed, not planned, they are remote, but these are the longest lasting. Often they are the opposite of the stated objectives. So remote, unintended, unexpected impacts are very often more important and more lasting and more dramatic than the short term intended, measured ones.” Aid consultant, Honiara
How widespread is our myopic focus on our intended results? A recent Netherlands Foreign Aid IOB study found unintended effects were an evaluator’s blindspot as across 664 evaluations over 20 years, “The ‘text miners’ found that only 1 in 6 IOB evaluation documents pay attention to unintended effects.” This dearth of attention to all the other things happening in projects led to 10 micro, macro, meso, and multiple level effects, from negative price effects such as food aid on local food producers or nationalist backlash to Afghan projects to positive effects (they found 40% of projects had this) such as a harbor built happening to expand beach tourism as well.
But if we don’t look for such effects, we don’t know the true impact of our aid programming. We also don’t honor the breadth of people’s lives rather than just as narrow ‘aid beneficiaries’ (ugh), not even honoring them with the terms’ participants’ much less ‘partners’ in their own development).
In our ex-post work, we find a wide array of ways ownership and implementation of activities is done after donors leave and without additional or with different resources, capacities, and partnerships. Taking emerging outcomes and impacts examples from a different Niger project, and one from Nepal:
1. Partnerships & Ownership: Half of the members of the all-women Village Banks reported helping one another deal with domestic disputes and violence. (Pact/Nepal)
2. Capacities: Trained local women charged rates to sell course materials onward (PACT/Nepal)
3. Ownership: Participants valued clinic-based birthing and sustained it by introducing locally-created social punishments and incentives (CRS/Niger)
4. Resources: New Ministry funding reallocated to sustain [health] investments, and private traders generated large crop purchases and contracts (CRS/Niger)
The assets and capacities we bring to help people and their country systems help only a sliver of their lives, and often in unexpected ways that sadly we aid donors and implementers don’t seem interested in. There are other puzzle pieces to add…
Let us not forget, as a Sustainable Development Goals Evaluation colleague said in 2017 on a call:
I am sure you, my dear colleagues, have reams of similar findings from your fieldwork. Please share yours!
by Jindra Cekan | Jul 15, 2021 | Accountability, Aid effectiveness, climate change, Covid, Financing gap, Impact Guild, International aid, Monitoring and Evaluation, post project evaluation, Private sector, SDGs, Sustainability, Sustainable development, Uncategorized
Sustainable Development Goals (SDGs), Funding and Accountability for sustainable projects?
What are Sustainable Development Goals? ” the United Nations adopted the new post-2015 development agenda. The new proposals – to be achieved by 2030- set 17 new ‘sustainable’ development goals (SDGs) and 169 targets. Some, like Oxfam, see the SDGs as a country budgeting and prioritization as well as an international fundraising tool. They cite that “government revenue currently funds 77% of spending…aligned with government priorities, balanced between investment and recurrent and easy to implement than donor-funded spending…” National investments are vital, but how much has the world used the SDGs to target investments and foster sustainable results?
Using results data such as that of the sectoral SDGs, countries can also ensure accountability for the policies implemented to reduce global and local inequities, but we must learn from the data. Over halfway to the goal, data is being collected, and while there is robust monitoring by countries who have built their M&E systems, other countries are faltering. “A recent report by Paris21 found even highly developed countries are still not able to report more than 40-50% of the SDG indicators” and “only 44% of SDG indicators have sufficient data for proper global and regional monitoring”. Further, there is very little evaluation or transparent accountability. Some of the data illuminate vitally need-to-know-for-better-programming. SDG data shows good news that Western and Asian countries have done better than most of the world 2015-19… but there is a lot of missing data while other data shows staggering inequities such as these:
- In Vietnam, a child born into the majority Kinh, or Viet, ethnic group is three and a half times less likely to die in his or her first five years than a child from other Vietnamese ethnic groups.
- In the United States, a black woman is four times more likely to die in childbirth than a white woman.
So are we using the SDG data to better target funding and improve design? This is the kind of evaluative learning (or at least sharing by those that are doing it :)) that is missing. As my colleague and friend Sanjeev Sridharan writes on Rethinking Evaluation, “As a field we need to more clearly understand evaluation’s role in addressing inequities and promoting inclusion” including “Promoting a Culture of Learning for Evaluation – these include focus on utilization and integration of evaluation into policy and programs.” How well learning is integrating is unknown.
As a big picture update on the progress of the Sustainable Development Goals (SDGs) in 2021, with only nine years left to the goal: It’s not looking good. The scorecards show COVID-19 has slowed down or wiped out many achievements, with 100 million people pushed into extreme poverty, according to the IMF. Pre-Covid, our blog on sectoral SDG statistics on health, poverty, hunger, and climate, was already showing very mixed results and a lack of mutual accountability.
The private sector is ever-being pushed to fund more of such development costs, only marginally successfully, as public sector expenditures are squeezed. Yet the G20 estimates that $2.5 TRILLION is needed every year to meet the SDG goals. As we have seen at Impact Guild, the push to incentivize private commitments is faltering. “To ensure its sustainability, the private sector has specific interests in securing long-term production along commodity supply chains, while reducing their environmental and social impacts and mitigating risks… The long-term economic impacts of funding projects that support the sustainability agenda are, thus, clearly understood. However, additional capital needs to flow into areas that address the risks appropriately. For example, much remains to be done to factor climate change as a risk variable into emerging markets that face the largest financing gap in achieving the SDGs.” Further, if decreased funding trends continue, by 2030, at minimum 400 million people will still live on less than $1.25 a day; around 650 million people will be undernourished, and nearly 1 billion people will be without energy access. So we’re not meeting the SDGs, they’re being derailed by COVID in places, and we aren’t beginning to cost out the need to address climate change and its effects on global development…. so now what?
From: https://www.g20-insights.org/policy_briefs/incentivizing-the-private-sector-to-support-the-united-nations-sustainable-development-goals/
To ensure that giving everyone a fair chance in life is more than just a slogan; accountability is crucial. This should include a commitment from world leaders to report on progress on “leaving no one behind” in the SDG follow-up and review framework established for the post-2015 agenda and for the private sector to loudly track their investments across the SDGs. For as The Center for American Progress wrote, money and results are key: We must “measure success in terms of outcomes for people, rather than in inputs—such as the amount of money spent on a project—as well as in terms of national or global outcomes” and that “policymakers at the global level and in each country should task a support team of researchers with undertaking an analysis of each commitment.”
A further concern. While we seem to measure the statistics periodically and see funding allocated to SDG priorities, but there are few causal links drawn between intensity in investment in any SDG goal and sustained results. To what degree are the donations/ investments into the SDGs linked to improvements? Without measuring causality or attribution, it could be a case of “A rising tide lifts all boats” as economies improve or, as Covid-related economic decline wiped out 20 years of development gains as Bill Gates noted last year. We need proof that trillions of dollars of international “Sustainable development” programs have any sustained impact beyond the years of intervention.
We must do more evaluation and learn from SDG data for better targeting of investments and do ex-post sustainability evaluations to see what was most sustained, impactful, and relevant. Donors should raise more funds to meet needs and consider only funding what could be sustained locally. Given the still uncounted demands on global development funding, we can no longer hope or wait for global mobilization of trillions given multiple crises pushing more of the world into crisis. Let’s focus now.
by Jindra Cekan | Jun 29, 2021 | ex-post evaluation, Norway, OECD, Project cycle, Sustainable development
Donors often talk about #SustainableDevelopment, and we know they try. But do they return #expost to #evaluate it? Here are lessons from #Nordic (#Norway, #Finland, #Sweden & #Denmark) and the #Netherlands in our #ValuingVoices #WhitePapers. Let us know what you think… Thanks, Jindra & Preston Stewart..
The document(s) cover:
1) the search process,
2) how ex-post evaluations are defined and categorized,
3) what was done well by each country’s ex-posts,
4) sustainability-related findings and lessons, and
5) what M&E experts in each country can improve on ex-post evaluation practices.
6) Case studies of the Netherlands, Norway, and Finland
One big finding is that there were only 32 evaluations that seemed to be ex-post project, and only 16 of them actually were at least 2 years after project closure.
Executive Summary/ Abstract of the set of White Papers
Ex-post completion or ex-post project evaluations are surprisingly rare given widespread commitments to ‘sustainable development’. Given the well-known commitment of the Nordic countries of Norway, Finland, Sweden, and Denmark as well as the Netherlands, Valuing Voices, a consulting firm specializing in ex-post project evaluation, chose these countries as case studies. We hired an intern from Harvard College, who searched the national databases (in English only) and presented these results. The findings were surprising. There were far fewer ex-posts in total than expected. Those that were included a surprising number that was co-evaluated alongside final evaluations, although final evaluation illuminates relevance, effectiveness, efficiency (and the new OECD criteria of coherence), while ex-post project evaluation illuminates sustainability and impact two or more years after closure. There were lessons for Ministries about the searchability as well as the quality (and in the case of Denmark and in part, Sweden), the dearth of ex-posts at all. Overall, the recommendations for the five countries researched for this paper include better and more standardized definitions of ex-post project, greater absolute numbers done, transparent sharing of those done in publicly searchable manners, methodologically clear comparisons (baseline and midterm), and clarity in differentiating different evaluations.
There are caveats to these papers. This research was privately by Valuing Voices so the samples featured from the five countries were limited to public sources about ex-post and ex-post evaluations. While we reached out to Ministry evaluative staff in all countries, only two made themselves available for consultation (the Netherlands and Finland) and provided a handful of additional ex-post evaluations. This paper focuses on what such research yielded, not definitive findings of programs or multi-year country strategies that are funded for 20-30 years continuously, nor projects funded by country-level embassies which did not feature on the Ministry site. We focus on project bilateral project evaluations, not multilateral funding of sectors. We also presented these findings in a 2021 webinar, during which we received input that Sweden’s EBA has a (non-project) portfolio of ‘country evaluations’ which looked back over 10 or even 20-year time horizons.. Still, we found very few ex-post evaluations at EBA (see cite in Search, below). No input was received by Norway or Denmark.
There are four papers in this series combined in this White Paper. First, the search across databases of all five Nordics (Norway, Finland, Sweden, and Denmark) and the Netherlands (pages 1-21). This is followed by country-specific, detailed, papers for those with ex-posts: The Netherlands (pages 22-37), Norway (pages 38-47), and Finland (pages 48-59). We look forward to your feedback. Jindra@ValuingVoices.com for the Valuing Voices team.
Let us know what you think… Thanks, Jindra & Preston Stewart
by Jindra Cekan | Jun 4, 2021 | Accountability, Africa, CRS, ex-post evaluation, Exit strategies, Faster Forward Fund, INGOs, Kenya, Local Participants, NGOs, SDGs, Sustainable development
Who is accountable for the ‘sustained development’ of those who suffer, and for how long?
Some of the rarely discussed myths of ‘sustainable development is that the aid we donors and implementers bring will help everyone, and that recipient governments can take over once donors leave and that what we left when the project ended was sustainable. The fact that our ‘global development aid’ helps a small fraction of all those who have equally ‘worthy’ needs in the countries we target is unspoken, as is the fact that these projects aren’t often built to sustainably withstand shocks such as climate changes bearing down now. As evaluator Michael Quinn Patton writes, “Effective programs… create islands of protected effectiveness in a sea of need and suffering… [we must] assess sustainability over time for Adaptive Resilience.”[1]
More often, countries need to be: a) so poor, b) so ‘fragile’, or c) so geopolitically important to warrant our aid. The fact that most USAID and EU bilateral funding (not necessarily multilateral funding to IMF, World Bank, African or Asian or other Development Banks) is focused on fragile/ war-torn or the strategically important countries edges out what is left for the impoverished people of the world. Most of our taxpayers have little idea of this and believe it is needs-based. So, many of us assume that: d) our mostly short-term aid is either only the brief support that they need (the ‘shot in the arm school’) as might be true of emergency or humanitarian aid post-emergencies or e) the results will be so excellent as to spontaneously spread (scale) ‘forever’ that no more aid will be needed or f) that national governments – not donors- need to carry the accountability for sustained improvements forward. Yet those stakeholders in poor or fragile countries (governments, non-profit NGOs) often have the weakest capacities to sustain results. Rutere Kagendo, a fellow Kenyan on the Valuing Voices team, wrote a moving blog about who ends up tasked with sustaining project activities and results: communities, especially women.
This is in spite of the fact that ex-post project evaluations done 1+ to 30 years after completion who asked participants and partners about sustainability, show ‘mixed’ results, namely that results fall off as early as 2 years after closure, by between 10-100%. Rarely do ex-posts have results improve or be as sustained as we assume. Some project activities do show somewhat lasting results, ranging from those that provide credit to those generating agriculture. Mostly lackluster results are because global development projects aim for success by closure over sustainability over decades by exiting better with accountability to our participants and partners over time. We do not design what can be locally sustained. Imagine how much would be saved if we did!
Jindra Čekan/ova of the USA/ Czech Republic and Peter Kimeu of Kenya offer our perspectives about who is accountable for aid, and for how long.
Jindra
I have worked for international NGOs (e.g. Catholic Relief Services, the international division of the American Red Cross, large INGOs such as CARE, World Vision, Lutheran World Relief, and others), including bilaterals (USAID) and foundations (the Bill & Melinda Gates Foundation, Aga Khan Foundation) for over 30 years. Most of our projects strove to fulfill objectives of the grants and were successful. Yet rarely did we ask ‘for how long?’ At one point, I worked for a huge International NGO with a program that had been feeding 50,000 West African children breakfast for 30 years. This was part of a bilateral aid education support + ‘safety net’ program. Many studies, including this terrific one from the UN’s World Food Program, show that such school feeding improves “school participation (enrolment, attendance, completion) and learning (scores on cognitive, language and mathematics tests)… [and] decrease child marriages, etc.“
Clearly, they do good. Yet the donor suddenly wanted proof that the effects of this long-term aid had improved national GDP rather than just the nutritional and learning outcomes of assisted individuals, otherwise, the project would be canceled. Letters from the impoverished country– including the country’s president and cabinet -stating they themselves had benefitted from the program which in turn led them to great educational outcomes and leadership had some effect. Yet without lobbying from the donor country’s agriculture and food aid industry that cutting food aid exports would harm them, it would have been canceled. Did they care about the schoolchildren’s learning or only providing outlets to US agricultural surplus producers? Was 30 years too long to keep helping? How do we have these discussions as equal partners? As evaluator Zenda Offir notes regarding the SDG’s No One Left Behind, “the burden of supporting and sustaining a majority of ‘leaving no-one behind’ efforts fall inevitably on many of the poorest (low-income) countries in the Global South. The problem is that they cannot afford it, nor can they sustain it. It will therefore be unfair to hold such countries accountable for ‘leaving no-one behind’ strategies. “
This brings up the questions of sustainability and accountability ‘to whom’ and ‘for how long’? You may have other questions, including ‘why’ and ‘how we know, which I look forward to addressing, but for now, these two are the focus of this blog. While 65% of Americans favor foreign aid, believing we spend up to 25% of our GDP abroad, the US spends just over 1%. Most US aid goes to the fragile and geopolitically strategic: “More than two hundred countries receive U.S. aid. It disproportionately goes to a few, however, with the top five all receiving over $1 billion per year as of 2016: Iraq ($5.3 billion), Afghanistan ($5.1 billion), Israel ($3.1 billion), Egypt ($1.2 billion), and Jordan ($1.2 billion).” In Europe, only 3 countries met the OECD goal of giving 0.7% of GNI: Norway, Sweden, and Denmark while the Czech Republic was at 0.13%, just below the USA 0.16%(2019). A fascinating measure of ‘commitment to development” (CDI) looking at the ‘quality of aid’ found that in the Czech Republic aid performance was very poor: “Adding up both quality-adjusted aid (95.7 million USD) and quality-adjusted charity induced by public policy (1.1 million USD), we arrive at 96.8 million USD for 2009 which amounts to 0.054% of GNI. Translating the percentage onto the standardised CDI scale the Czech Republic… [has] the third least favorable aid policy towards developing countries among DAC countries… [and] Aid allocation is not primarily focused on low-income countries” which is in part explained by the recent shift from aid recipient to aid donor (Syrovatka and Krylova 2012). Even, worse now, donors and international NGOs distribute aid (especially what is left for the poor countries of the world) that is annually allocated, but as the pandemic has led more spending to be domestic, aid to the poorest has decreased among almost all donor countries, bad news as Covid-economic downturns continue and climate change ramps up.
Is there proof there is no more need for aid to places or people? We do not know as we return to evaluate the sustainability after projects close far less than 1% of the time. Mostly this is because we assume that the recipient governments have ‘taken over’, that there is funding from elsewhere, or that the communities and organizations helped have become as resilient as to keep up the good work themselves. Yet the ex-post evaluation data does not bear this out. Few such evaluations are done or done well and many assumptions of positive trajectories are unproven. Donors and INGOs want to help, must leave after money is spent, and assume the best. Local participants implement the projects but they do not design or lead their implementation, which limits continuation after donor support exits.
So who is accountable to the poor whom we help? Peter comments on that from the perspective of the CEO of a local Kenyan NGO targeting 15,000 farmers.
Peter
I have over forty years of experience in development; 8 years lead in Community Initiated (Harambee) High schools, 35 years with Catholic Relief Services in Emergency Relief, Sustainable Development and Justice and Peace Programs, and currently 6 years with the (Kenyan) county-devolved sustainable development. I am the Founder and CEO of Decent Living Institute of Organic Farming promoting avocado farming, aquaculture, and apiculture for improved decent livelihoods. My early life as a young boy makes me a living witness of a life in deep poverty, which the New York Times featured.
The question ‘who is accountable for sustained development’ and ‘for how long’ has an assumption that it is possible to attain sustainable development without the continued involvement of those who suffer. I don’t think so. Sustained development occurs as a process to a transformed situation from abject poverty, a condition of want without the capacity to satisfy even the most basic needs, a position of lacking continuously leading to untold suffering and living in dehumanized conditions for the sufferer and the generations to come to the desired decency of fulfilled living. Living as a pauper in my first 30 years of life, having been born in a paupers’ family, I accepted the conditions of poverty and hunger as a way of life. After all, you know nothing better and when you see wealth around you, it is meant for the lucky few, and not for you. The situation limits the poor to survival conditions, eating from hand to mouth and everything is left to luck.
Aid to the poor would make sense if it is used as a catalyst to motivate and enable the poor depart from the circle of poverty (the poor giving birth to more poor) and is able to sustain the conditions of being above the poverty line of US$2 a day. Such aid would enable them to have enough to take care of their daily basic needs and create wealth without falling back below the poverty line repeatedly, for generations to come. The impact measure for aid should therefore be participatory learning from and measuring the extent to which success is sustained documenting representative success stories by participants who have left the circle of poverty sustainably. Such would include ‘in the past I couldn’t to find enough to eat occasionally slept without food and now my family has no idea of how it feels like to be hungry.
Unfortunately, the manner of delivering aid is seen as pure luck by the targeted poor for it comes without involving the poor as to strategically plan long-term impact that they can sustain. The aid donors and implementing agencies will target a given county, while the identification of targeted community cluster location for aid will be influenced by either by powerful persons from the locality or larger numbers in the public participation, so those with greater authority or louder voice will take the day. The decision on who will participate in the project finally will be determined by the same criteria and not the poverty levels. One example is the aid fund for COVID-19 response in Kenya which was distributed to the well connected to persons of authority and not to those who championed the control of the coronavirus. A decision was made at the county governments to disperse one million Kenya shillings to every cluster of villages to pay the youth for engaging in communal work such as community road works, terracing a degraded land, or even constructing an earth dam and paid per piece work completed – termed employment -to cushion the youth who have lost job opportunities due to the COVID -19 effects. A million shillings in a cluster of ten villages would perhaps engage 100 young people for a week earning Kenya shillings 500 (US$5) a day or $30 a week. The rest of the money – 50% of the total or more – does not go for wages as planned but is used to cover the management of the program by the county officials. The youth will spend this money like it is good luck for it is too little to ever think of the future investments.
However, the same amount is what it costs to support a member of the self-help group and collective community-led development in our Decent Living NGO per family of a vulnerable child to grow a vegetable garden, keep six chicks and grow three Hass avocado seedlings. Further, the participants commit to support another poor family with six chicks in a years’ time. From the onset, the poor are involved in ‘planning in advance’ to help others. Their developing vision is guided by the long-term impact they hope for, such as the family economic boost that will cover the full cost of schooling, medical expenses, and family meals, clothing, and shelter for all the children including the most vulnerable. Other long-term indicators will be the percentage of poor families that are above the poverty line meeting the family basic needs sustainably.
I see the role of the aid donor as to holding the aid receiver (local government and recipient communities) to their goals of sustainable development and to account for the funds given by reaching their goals and targets that must be time-bound. The aid receivers are also responsible to account for their aid distribution to their intermediary implementing partners (often local non-profits/ NGOs) to meet their targets, goals and should track expected and measurable long-term outcomes within three years after the closure of the project. This means to deliver not only the aid funds but also through the funding the systems established or improved at the conception of the project should be accounted for during the project implementation period and will be impacting long term results transforming the community to the desired state long after the project activities.
It could be building sustainable infrastructure for long-term support to the poor. The sustained impact would then be numbers of poor that have transformed their poverty and created wealth through the developed infrastructure in an intergenerational, long-lasting way that could be measured in later years. Sadly, most of the aid givers do not see their role beyond the performance short-term outputs such as trainings given or outcomes leading to a change in farming practices, like the deliverables for the specific objectives in an agriculture project. Hence the success of the project is defined by these short-term indicators that measure outputs such as the target number reached with food aid, or even some changes in practices leading to improved yields, but once the project ends, all tracking of results end. The national stakeholders are – or should be- responsible to demonstrate how the results of the project will be assumed by the community’s self-help groups so that the impacts become intergenerational. For it is vital to see that the project does not end with the implementing agencies. It is not only short-sighted by aid donors to believe that it ends, but national stakeholders are absconding their responsibility of accountability to the long-term impacts that are related to relationships and behavior change sustainably when they do not sustain them.
The UN’s ‘Sustainable Development Goals’ are merely a dream for most poor until the individual struggling with conditions of want is able to take steps towards permanent solutions for themselves and their future generations. It takes an oppressed dreamer (the poor with empty stomachs) who believes a change is possible to demand accountability. It also takes a progressive facilitator (donors and national stakeholders) who believes in creating enabling conditions for the oppressed to succeed. Both the oppressed dreamers (project participants, local implementing NGO agencies and the progressive facilitator (donor) are accountable to the transformed conditions. For the ‘sustained development’ to occur it must be intentionally dreamed of by all parties engaged in the process of development.
I dream of “a just world where everyone is fully a participant and celebrates sustainable development for all” wrote Pope Francis in his 2015 Laudato Si encyclical. He calls for all humanity to take care of our Mother Earth and in return, she will provide for all, including addressing issues of global warming. In my world dream, I see a time when a transformation of the sufferer from distressful and oppressive conditions of living is eased by putting future dreams into action, for those who suffer with deprivation today and are thirsty for change. I wish to make reference to a story told in the bible Jesus meeting a blind beggar (Mark 10:51) shouting to Jesus for help. Jesus asked the beggar to identify what type of aid he needed. And the blind beggar’s request for the power to see was heard and his sight was fully restored, emancipating him from the bondage of begging. We are told he transformed from a beggar into a disciple of Jesus. The transformation of conditions to sustainable options starts with the bilateral donor engaging a poor government to undertake a particular development agenda that in return facilitates its citizens to enjoy sustained development. The donor government should hold the recipient government responsible and accountable of delivering sustainable options for its citizen as per the grant agreements with evaluation two to three years after the project closure.
The poor who may be targeted with the aid may seem passive, not having been involved right from the beginning of design, and may have limitations of identifying what to ask for, perhaps those with intermediary implementing INGOs may be aware of how well what is being offered can meet their needs. Setting up the appropriate structures, they may dictate and demand sustained development options for themselves and those who are suppressed in poverty. The major issue is that most often the victims of poverty are never engaged in aid’s design and only implement what is offered. The situation creates room for corrupt national governments, INGOs, and NGOs to make quick money. The donors should hold recipient countries and INGOs accountable for tangible results toward the Sustainable Development Goals indicators for every grant in aid for as long as it takes, not just reporting at the national level.
I see a world where what matters most, is how engaged those who have empty stomachs are in the development aid agenda, and how the aid is administered and accounted for themselves and the neighboring suffering households. That development is all about a sustained transformation for empty stomachs of our project participants, their immediate neighbors, their children, their husbands, and their fathers/in law and mothers /in law, their brothers/ in law and sisters/ in law. It is about a better living standard of their neighbors who lends salt and water, the generosity of their firewood friends, those neighbors who will never turn down an opportunity to offer help no matter what. If these impacts and long term outcomes are not evaluated and accounted for, those who suffer poverty will always consider projects as myths of ‘sustainable development’ and the aid provided by bilateral, multilateral donors or through INGOs/NGOs as beneficial to the lucky few, while recipient governments and participating communities and their future generations have no sustainable impact results.
Your thoughts?
[1] MQ Patton in New Directions for Evaluation “Transformation to Global Sustainability: Implications for Evaluation and Evaluators, 2019 (link inaccessible)
