On my way to present at the European Evaluation Society’s annual conference, I wanted to close the loop on the Nordic and Netherlands ex-post analysis. The reason is, that we’ll be discussing the intersection of different ways to evaluate ‘sustainability’ over the long- and short-term, and how we’re transforming evaluation systems. The session on Friday morning is called “Long- And Short-Term Dilemmas In Sustainability Evaluations” (Cekan, Bodnar, Hermans, Meyer, and Patterson). We come from academia as professors, consultancies to International organizations, International/ national non-profits, and our European (Dutch, German, Czech), South African, and American governments. We’ll discuss it as a ‘fishbowl’ of ideas.
The session’s abstract adds the confounding factor of program vs project versus portfolio-wide evaluations all-around sustainability.
Details on our session are below and why I’m juxtaposing it to the Nordic and Netherlands ex-posts in detail, comes next. As we note in our EES ’22 session description, “One of the classic complications in sustainability is dealing with short-term – long-term dilemmas. Interventions take place in a local and operational setting, affecting the daily lives of stakeholders. Sustainability is at stake when short-term activities are compromising the long-term interests of these stakeholders and future generations, for instance, due to a focus on the achievement of shorter-term results rather than ensuring durable impacts for participants… Learning about progress towards the SDGs or the daunting task of keeping a global temperature rise this century well below 2 degrees Celsius above pre-industrial levels, for instance, requires more than nationally and internationally agreed indicator-systems, country monitoring, and reporting and good intentions.”
But there are wider ambitions for most sustainability activities undertaken by a range of donors, policy actors, project implementers, and others: Sustainability “needs to span both human-social and natural-ecological systems’ time scales. Furthermore, long-term sustainability, in the face of climate change and SDGs, demands a dynamic view, with due attention for complexity, uncertainty, resilience, and systemic transformation pathways…. the need for a transformation of current evaluation systems – seeing them as nested or networked systems… Their focus may range from focused operational projects to the larger strategic programmes of which these projects are part, to again the larger policies that provide the context or drivers for these programmes. Analogue to these nested layers runs a time dimension, from the short-term projects (months to years), to multi-year programmes, to policies with outlooks of a decade or more.”
When Preston did his research in 2020-21 which I oversaw, we focused on the projects precisely because that is where we believe ‘impact’ happens in a measurable way by participants and partners. Yet we found that many defined their parameters differently. Preston writes, “This paper focuses on what such research [on projects evaluated at least 2 years post-closure] yielded, not definitive findings of programs or multi-year country strategies that are funded for 20-30 years continuously, nor projects funded by country-level embassies which did not feature on the Ministry site. We focus on project bilateral project evaluations, not multilateral funding of sectors. We also …received input that Sweden’s EBA has a (non-project [not ex-post] portfolio of ‘country evaluations’ which looked back over 10 or even 20-year time horizons…
So we present these compiled detailed studies on the Netherlands, Norway, Finland, Sweden, and Denmark for your consideration. Can we arrive at a unified definition of ‘sustainability’ or imagine a unified ‘sustainability evaluation’ definition and scope? I hope so, will let you know after EES this week! What do you think, is it possible?
Reblogged from the journal EVALUATION,Volume: 28 issue: 1, page(s): 7-35, Article first published online: January 24, 2022; Issue published: January 1, 2022
“Leading evaluation practitioners were asked about lessons from the recent 26th Conference of the Parties (COP26) for evaluation practice. Contributors emphasize the importance of evaluating equity between rich and poor countries and other forms of climate injustice. The role of the evaluation is questioned: what can evaluation be expected to do on its own and what requires collaboration across disciplines, professions and civil society – and across generations? Contributors discuss the implications of the post-Glasgow climate ‘pact’ for the continued relevance of evaluation. Should evaluators advocate for the marginalized and become activists on behalf of sustainability and climate justice – as well as advocates of evidence? Accountability-driven and evidence-based evaluation is needed to assess the effectiveness of investments in adaptation and mitigation. Causal pathways in different settings and ‘theories of no-change’ are needed to understand gaps between stakeholder promises and delivery. Evaluators should measure unintended consequences and what is often left unmeasured, and be sensitive to failure and unanticipated effects of funded actions. Evaluation timescales and units of analysis beyond particular programmes are needed to evaluate the complexities of climate change, sustainability and to take account of natural systems. The implications for evaluation commissioning and funding are discussed as well as the role of evaluation in programme-design and implementation.”
Here is my article on sustainability, measurement and reporting:
“Promises made and promises unfulfilled: Focusing evaluations after COP26
Like many evaluators reading this, I am not a climate specialist but an international political economist, a Czech-American. Both my countries have polluted more than our fair share. Maybe like you, I feel responsible for those who polluted less but suffer more. Professionally, I focus on grassroots sustainability of ex-post project evaluations, including those funded by the Adaptation Fund, and consult on environmental, social and governance ‘impact’. I worry that aid impacts sustained through ingenious local efforts will not hold up to climate shocks for which our aid was not designed, and funding is insufficient.
Where to focus? Knowing what aspect of evaluation interests you, shapes which aspect of the COP26 juggernaut to examine. Results gaps between promises made versus actual change accomplished are an evaluator’s daily bread. Were I an evaluator of environmental processes such as deforestation, ocean acidification/ biodiversity, CO2 emissions, I could evaluate along the lines of the recent publication edited by Juha Uitto (2021) Evaluating Environment in International Development. Abel Gbala, an Ivoirien monitoring and evaluation expert, offers a range of roles evaluators take on, and two for climate change are
- Evaluator as a ‘judge’ (following Scriven) to investigate and justify the value of an evaluand, supported by both empirical facts and probative reasoning;
- Evaluator as ‘activist’, as argued by Bitar (2019), and Montrosse-Moorhead, et al. (2019: Chapter 3, 33) advocating for social justice and addressing the needs and interests of the vulnerable and disadvantaged.
Following the money and focusing on the centrality of justice and equity between rich and poorer/‘developing’ countries involves judging and being an activist with sharing results. This includes measuring how well the Global North has helped the Global South deal with the inequity in adapting to, mitigating or addressing the devastation of a range of climate change disproportionately caused by the Global North over two centuries. Notably, only small proportions of all financing get to indigenous and local communities (see Rainforest Foundation Norway, 2021; USAID, 2021). Evaluating to whom funding goes is vital for sustainable results.
Many promises are unfulfilled
The 2015 Paris Agreement (United Nations Framework convention on Climate Change, 2016) promised fewer climate-harming emissions, yet the 60 biggest banks have invested 3.8 trillion in fossil fuels (Project Regeneration, 2020). Paris signatories promised US$100 billion climate funding a year, but the COP26 showed massive shortfalls. Not only have an insufficient US$55–US$80 billion a year been given since 2013 (Timperley, 2021), but in a recent Financial Times article by Hook and Kao (2021), Amar Bhattacharya, of the Brookings Institute, stated, ‘In terms of real impact of climate finance, and efficacy across different donors, there has been no development impact or climate impact study done to date’. A German climate watchdog confirms a massive gap if and how US$80 billion in 2019 has been spent, noting, ‘The absence of a detailed, publicly available account of this financing . . . risks all sorts of omissions: donors mis-labeling their funding [as “significant” [impactful], or money being misspent, or an under-estimation of the true volume of money required’ (Subramanian, 2021). Evaluators and auditors are needed to confirm that funding was allocated, disbursed and had an impact on climate change needs.
Needs are tenfold more
India and African countries state they need US$1–US$1.3 trillion in finance by 2030 (Rathi and Chaudhary, 2021). This is not unreasonable, given that ‘developing’ countries ‘are currently shouldering approximately $70 billion per year costs of adapting to climate change’ themselves. The Global South also wants funds to be more evenly split between adaptation (now 25%) to help them deal with sea-level rise and extreme weather events and mitigation (now 75%) (Pontecorvo, 2021). Why the imbalance towards mitigation? Because mitigation is remunerative to investors, companies and banks, who offer loans for countries to switch to clean energy or sell ‘carbon-offsets’. As noted by Timperley (2021), ‘just $20 billion went to adaptation projects in 2019’ versus UN-estimated needs of US$300 billion. It is also essential to measure the effectiveness of finance once it arrives and help those in the climate field see how such investments’ efficacy can be improved.
Where to go to measure costs and finance? One priority for evaluators is to know where to look for data on finance and costs on sustainability and adaptation. This is spread over many national and international databases and reports, and across private and public institutions. Burmeister et al. (2019) have a useful table that summarizes the many finance sources that could be used by evaluators when trying to track actual expenditures and investments on adaptation.
Proof of promises is key
Oxfam’s Climate Finance Shadow Report 2020 (Carty et al., 2020) helps judges and activists see that while donors reported giving US$59.5 billion in 2017 and 2018, ‘the true value. . . may be as little as $19-22.5 billion per year once loan repayments, interest, and other forms of over-reporting are stripped out’. Eighty percent was primarily given as loans, and a further 50 percent of this was non-concessional, requiring higher repayments from emerging countries. In short, our climate ‘largesse’ is increasing their indebtedness. Another watchdog looks at the recipient side. Climate Governance by Transparency International (2021) traces in-country corruption of the funds received. The International Financial Reporting Standards Foundation’s International Sustainability Standards Board questions corporate ‘greenwashing’. Other evaluations remind multilateral and bilateral donors not to claim what they cannot substantiate. Aid promises ‘sustainable development’. Climate funds such as the GEF could be delivering, but Čekan/ová and Legro (2022) examined the GEF’s 2019 report claim that 84 percent were sustainable post-project. ‘Can We Assume Sustained Impact? Verifying the Sustainability of Climate Change Mitigation Results’ showed no proof of ex-post project fieldwork or research to substantiate it. Worse, ‘in the absence of sufficient information regarding project sustainability, determining post-project greenhouse gas emission reductions is not possible, because these are dependent on the continuation of project benefits following project closure’.
It is vital to monitor and evaluate the gaps between promises made and actual change. Gaps include between the finance needed by developing/poorer countries and what is delivered; provable measurements of the impacts and effectiveness of finance given; and the knock-on effects of support for climate action, including indebtedness. As evaluators, we need champions willing to listen, for no one has to listen to evaluators, but much like years of the climate-science IPCC, perseverance and public interest, plus our collective survival on the line, measurements increasingly matter and drive imperative change. Our planet, institutions and many promises and fewer results need all of us.”
I also encourage readers to see the other 13 authors” fascinating submissions in Evaluation post-COP26. Also, many thanks to Elliot Stern, editor, for his support and for making this issue open-access for global learning.
Follow our path on Ex-Post Sustainability and Resilience Seasonal Joy
Rarely do we get to teach, innovate, learn, and expand a hidden corner of one’s field. This is what is I am experiencing, and the Fund is sharing. Here is the Adaptation Fund’s ex-post project evaluation of sustainability and resilience path and progress in 2021.
I am quoting liberally and highlighting our work from the Adaptation Fund’s website where their commitment to learning from what lasts is clear. “Ex post evaluations are a key element of the AF-TERG FY21-FY23 strategy and work programme, originating from the request of the Adaptation Fund Board to develop post-implementation learning for Fund projects and programmes and provide accountability of results financed by the Fund. They intend to evaluate aspects of both sustainability of outcomes and climate resilience, and over time feed into ex-post-evaluation-informed adjustments within the Fund’s Monitoring Evaluation and Learning (MEL) processes.”
How are we defining sustainability’s path to evaluate it? Here is a flowchart from our training:
There are four phases from 0 to 3:
Phase 0 Foundational Review: Not only was this work preceded by months of background research on both evaluability of their young portfolio (e.g., under 20 of the 100 projects funded were closed at least three years, a selection criteria we had) and secondary research on evidence of ex-post sustainability evaluation in climate change/ resilience across the Adaptation Fund’s sectors.
Phase 1 Framework and Pilots Shortlist: Our Phase 1 report from mid-2021 provided an overview of the first stage of ex-post evaluations, outlining methods and identifying a list of potential projects for ex-post evaluation pilots from the Fund’s 17 completed, evaluated projects. The framework presented in the report introduced possible methods to evaluate the sustainability of project outcomes, considering the characteristics, strengths, and weaknesses of the Fund portfolio. It also presents an analysis tool to assess climate resilience, bearing in mind that this area is pivotal to climate change adaptation yet has rarely been measured.
The Phase one report on ex post project sustainability evaluation
Vetting and pilot selection, revised design for evaluating sustained outcomes related to resilience to climate change. Key aspects are: 1) Timing (3-5 years since closure or projects at least 4 years long within the last 5 years and seasonality matches the final evaluation) and 2) Good quality of implementation and M&E with measurable outputs and outcomes traceable to impact(s) and 3) Safety to do fieldwork re: Covid, civil peace, etc.
We (my so-clever colleagues Meg Spearman and Dennis Bours) introduced a new resilience analysis tool that includes consideration of the climate disturbances, the human and natural systems (and their nexus) affected by and affecting project outcomes. This includes five characteristics of resilience in the outcomes (presence of feedback loops, at scale, plus being diverse, dynamic, and redundant) and means/actions to support outcomes. Resilience can be identified via a clear summary of the structures (S) and functions (F) that typify Resistance, Resilience and Transformation showing where a project is and is moving towards. It is a typology of resistance-resilience-transformation (RRT) onto which the overall project can be mapped based on how actions are designed to maintain or change existing structures and functions. That was integrated into the Adaptation Fund resilience evaluation approach.
Phase 2 Methods Testing and Ex-post Field-testing: Training of national evaluators and piloting two ex-post evaluations per year includes selecting among these methods to evaluate sustainability ex-post plus the RRT and resilience measures above. In the first ex-post in Samoa’s “Enhancing Resilience of Samoa’s Coastal Communities to Climate Change” (UNDP) happening December 21, it is through qualitative evaluation of wall-infrastructure. The second, Ecuador’s “Enhancing resilience of communities to the adverse effects of climate change on food security, in Pichincha Province and the Jubones River basin “(WFP) has training completed and fieldwork should be from January 22, likely be of food security assets and methods TBD.
Phase 3 Evaluations continue, with MEL Capacity Building: Two more years of ex-post pilot evaluations (2 per year) with lesson informing integration into the MEL of the Adaptation Fund. We are already finding out lessons of rigor, of knowledge management, of unexpected benefits of returning years after closure, including indications of sustainability and resilience of the assets, with much more learning to come.
Innovations include “the relative novelty of climate change adaptation portfolios and the limited body of work on ex post evaluation for adaptation, it presents possible methods that will be piloted in field-tested ex post evaluations in fiscal year 2022 (FY22).” This includes piloting shockingly rare evaluation of oft-promised resilience. In the update to AF’s Board three months ago, it transparently outlined shortlisting of five completed projects as potential candidates for the pilots, of which two projects were selected for ex post evaluations. It outlined our process of co-creating the evaluation with national partners to prioritize their learning needs while building national capacity to assess sustainability and resilience of project outcomes in the field onward.
Also, training materials for ex post pilots are being shared to foster country and industry learning, focusing on evaluating projects at ex-post and emerging sustainability and resilience, as well as presenting and adapting methods to country and project realities.
The training material for ex-post pilots
The training had three sessions (which could not have happened without colleague Caroline’s expertise):
- Part A: Understanding ex-post & resilience evaluations. Introduce and understand ex-post evaluations of sustainability and resilience, especially in the field of climate change adaptation
- Part B: Discussing country-specific outcome priorities and co-creating learning with stakeholders. Discuss the project and its data more in-depth to understand and select what outcome(s) will be evaluated at ex-post
- Part C: Developing country-specific methods and approaches. Discuss range of methods with the national evaluator and M&E experts to best evaluate the selected outcome(s) and impact(s)
Overall progress can be seen in the document updating the AF’s board: A progress update on ex post evaluations (AFB/EFC.28/Inf.4)
So in 2018 I Wished for Sustained Outcomes to be explored and in 2019 I Wished again for more Ex-post Evaluation than Needles in Haystacks. In 2021 some of these wishes are becoming fulfilled! Seasonal Happiness for me is learning about resilience to climate change, diminished vulnerability and searching for proof of sustainability and emerging outcomes and impact(s) and I am grateful to the Adaptation Fund for its commitment to sustainability.
Accompany us on this path, cheer us on, and do your own ex-post sustainability and resilience evaluations! Happy holidays from the Czech Republic!
5 Ways to Foster Sustainability and Resilience to Climate Change
by Omar Abdou, M.A. and Jindra Cekan/ova, Ph.D
In the global climate change graphic, the blue are countries that contribute very few emissions to #climatechange. They are also those who suffer the most from climate change effects and who have most #globalaid projects which are most at risk of no longer being #sustained:
Omar Abdou, a food security and M&E expert from Niger, lives a vulnerability to climate change there is indisputable. Sahelien production systems suffer the full brunt of the negative effects. Indeed, the essential rural activities, namely agriculture, and livestock which occupy at least 80% of the population, are becoming more and more uncertain as they rely on two pillars: rainfall and the exploitation of natural resources (soil, vegetation, other water sources). But, in recent decades, we have witnessed a drastic reduction in rainfall. In pastoral zones, this decrease is reflected in the degradation of the vegetation cover and the deficits in livestock-vital fodder. For instance, the fodder assessment from 2000 to 2020 for the Tahoua Region has been in deficit for 15 years. More and more recurrent and frequent crises seriously affect the main productive capital of households, which is the herd.
Emergency sales of increasingly emaciated animals at a low price are growing to pay for consumer goods and to buy fodder to feed a few animals. These are kept for breeding after the crisis, but what used to be seasonal crisis (dry-season) has become chronic.
In agro-pastoral zones, agricultural production in the rainy season, which has changed its duration and strength, becomes uncertain to feed the household members. The harvest is often exhausted at most by the 6th month of the year for many households. Thus, there is no more food surplus to supply the area’s food needs where cereals are not grown. Suddenly, cereals become expensive and inaccessible for many families. Their coping strategies are diverse but not always effective. In pastoral settings, parents are forced to take their children out of school to go further south in search of food and pasture. Sometimes more sedentary agro-pastoralist farmers pledge or sell their land to pay for food. Those left behind who have nothing to sell, are forced to cut trees and sell the wood for charcoal, further degrading the environment. Unfortunately, Niger’s neighbors are now mostly at war, so these young men run the risk of being recruited by armed groups or are killed. In these conditions, there is serious instability where the means of production (natural and human) are sold for survival and overexploited natural resources decrease. Increasingly, it is practically impossible to talk about sustainability. It is, therefore, necessary to think of consistent and innovative projects to enable households to produce sustainably while preserving the ecosystem for future generations. Outlets such as migration of the able-bodied workers to neighboring countries to provide labor in the lean season work now but are unsustainable.
Jindra notes that 31 Oct the UN Climate Change Conference (COP26) begins. Climate science documents life-threatening effects such as Omar and billions of others are already experiencing around the ‘developing’ world. The IPCC Working Group report shows that “the world will probably reach or exceed 1.5 degrees C (2.7 degrees F) of warming within just the next two decades. Whether we limit warming to this level and prevent the most severe climate impacts depends on actions taken this decade. Only with ambitious emissions cuts can the world keep global temperature rise to 1.5 degrees C, the limit scientists say is necessary for preventing the worst climate impacts. Under a high-emissions scenario, the IPCC finds the world may warm by 4.4 degrees C by 2100 — with catastrophic results.” Erratic rainfall, greater heatwaves will affect 70% of all farmers and herders who rely on rain for their production and a new report on Adaptation to climate across West African shows aridity increasing, which must inform global development.
What can we do:
1- LISTEN AND ACT GLOBALLY: There are excellent resources to track demands that the Global North divest and stop subsidizing fossil fuels. For instance, “In 2017/18, the G7 collectively spent more on fossil fuel finance than on climate finance, according to analysis by Oil Change International. They allocated nearly $40bn that year.” Hypocrisy abounds as this Climate Change News article goes on to note that “Since the Paris Agreement was signed, G20 countries’ export credit agencies have provided 14 times as much support for fossil fuels as clean energy “to $64 billion a year, and investors still prioritize profit over planet & people.
2- LISTEN AND PRIORITIZE GLOBAL SOUTH VOICES: While funding demands for over $100billion to LDCs is a demand of COP26, the need to support local research and activist organizations who show measured results is mostly missing (more from Sweden here). Other resources include WRI’s Allied for Climate Transformation by 2025 podcast series among the most Climate Change vulnerable nations. “ACT2025 consortium to ensure voices from countries most exposed to climate change are heard, empowered, mobilized, and adequately supported in international climate negotiations.“ The series includes how countries such as Niger adapt to climate change. If nothing else, consider the tragedy of the loss of $1.5 billion in aid invested in Niger, not to mention the ensuing suffering global Northern consumption is unleashing on the global South.
3- DESIGN CLIMATE SMARTLY: Global Development needs to do much more climate-smart programming as 50 years of investments in foreign aid to ‘developing’ countries has already started to be threatened. What can be done? From sequestering the carbon content of soil via farming to fostering climate-smart agriculture, which, according to the UN’s FAO means “agriculture that sustainably increases productivity, enhances resilience (adaptation), reduces/removes GHGs (mitigation) where possible, and enhances achievement of national food security and development goals”. Importantly this involves productivity plus adaptation and mitigation against changes wrought by climate change.
4- MEASURE SUSTAINABILITY: Exciting measurements are beginning! Scotland is valuing its nature, and there are calls for The Rights Of Nature In Evaluation Of Environmental Sustainability. Jindra is consulting to a wonderful team at the Adaptation Fund. We are evaluating what could be sustained, and what withstood climatic changes is ex-post project sustainability and resilience evaluation. The Adaptation Fund on evaluating projects’ longer-term (3-5 year ex-post project closure) sustainability and, importantly their resilience to climate change. We are piloting these rare kinds of evaluations in six countries, starting with Samoa and Ecuador in 2021. As the Fund has funded over 100 projects in 100 countries, with almost 30 million participants, we have a rich array to choose from. Such a learning opportunity about what lasts and has withstood disturbances is vital. As the Fund notes: “Climate change is predicted to greatly affect the poorest people in the world, who are often hardest hit by weather catastrophes, desertification, and rising sea levels, but who have contributed the least to the problem of global warming. In some parts of the world, climate change has already contributed to worsening food security, reduced the predictable availability of freshwater, and exacerbated the spread of disease and other threats to human health. Helping the most vulnerable countries and communities is an increasing challenge and imperative for the international community, especially because climate adaptation requires significant resources beyond what is already needed to achieve international development objectives.”
5- TAKE PERSONAL RESPONSIBILITY: For 8 years, Valuing Voices has pushed back to evaluate the ‘Sustainable Development” we have promised for 50 years. We must be accountable primarily to our participants and partners, rather than donors, and only by listening to these country nationals, we can learn what was sustained not, why or why not, and what emerged from their efforts. In 2019, we wrote all of us are accountable to improve the SDGs. It is wonderful to have climate change leading organizations invest in ex-post evaluation that adds resilience, prioritizing organizational learning and accountability. We all must increase our awareness that curb our consumption, for the sake of Niger, small island states, and many other parts of our planet can no longer stand our business-as-usual. Limiting climate change begins with us, from changing our food consumption to consumer choices to political advocacy to seeing the Earth as indivisible from us and us from the Earth. Let us start to invest in global sustainability today.
Do you see any bright spots on the horizon? Do share!
Aid providers: More puzzle pieces, including unexpected outcomes; ours is not the whole picture
When we did our first ex-post evaluation/ delayed final evaluation in 2006 in Niger for Lutheran World Relief (LWR) funded by the Bill & Melinda Gates Foundation (pg75 on), we found all sorts of unexpected/ unintended outcomes and impacts that far outweighed the original aid’s expectations. The project measured success by the livelihoods rebuilding post-drought ravaged sheep herds and water points for them. Instead, while the LWR aid was beautifully based on ‘habbanaye,’ a pastoralist practice of lending or giving small-stock offspring to poorer family members (and was expanded to passing on animals to poorer community members), and results showed a majority of the poor benefitted, our respondents showed it was far more nuanced. We aid providers (and our expectations) are only a part of any ‘impact’, which needs to be defined by the communities themselves:
- While many families benefitted from the sheep, enabling some young boys to shepherd several at a time, it turns out the poorest chosen by the communities were not necessarily ones who lost small livestock during the drought but were, in fact, the ultra-poor who had never had them. Therefore, proof of successful ‘restocking’ post-drought left these poor who were helped the most, out, as they were… unexpected;
- Holding onto the donated sheep was not as important an indicator for some: one woman told us that selling her aid-received sheep to buy her daughter dowry to marry a wealthier husband was a far better investment for their financial future than the sheep would have been. Our main measure of success was not nuanced enough;
- The provision of water through well-rehabilitation and building in the five villages was a vital resource. Women reported they saved 8 hours every two days by having potable water in their villages. Before then, they spent three hours walking each way to the far-off well and waited two hours to fetch 50l water, which they head-carried back. With the well in place, they generated household income through weaving mats, cooking food for sale, etc., which amounted to as much as 20% of increased household income – a boon! Also, having both time and water access enabled them to bathe themselves and their children, make their husbands lunch, and make their mothers-in-law tea, which led to far more household harmony. No ‘impact’ measures outside of livestock and water were included or could be added;
- Finally, the resulting show-stopper: in the last village where we interviewed women participants, they said that the groups of fellow recipients were a boon for community solidarity across ethnic groups. In their meetings, thanks to the sheep, water, and collective moral support, they said the conversations turned from conflict to collaboration, and best of all, women reported, “our husbands don’t beat us anymore.” Peace among ethnic groups, within and between households was completely unexpected.
Such a highly unexpected outcome would fall under #2 and #4 of the Netherlands study below. Unfortunately, the Foundation seemed less interested in these unexpected but stellar results. Yet at the same time I have empathy for their position, as so many of us in global development want to help solve problems, and demand proof we have.…. so we can leave and help others, equally deserving. Taking complexity into account, seeing lives in a wider context where our aid can be helping differently or even harming makes garnering more aid hard.
As the brilliant Time to Listen series by CDA showed, aid intervenes in people’s lives in complex ways and we need to listen to our participants and partners who always share more complex views than our reports can honor..A few of hundreds of quotes of 6000 interviews, this from the Solomon Islands:
Some appreciate the aid as it is given:
“People in my village are very grateful for the road because now with trucks coming into our village, the women can now take their vegetables to the market. Before, the tomatoes just rotted in the gardens. Tomatoes go bad quickly and despite our attempts in the past to take them to the market to sell, we always lost.” Woman from East Malaita
But for others there are great caveats:
“Donors should send their officers to Solomon Islands to implement activities in urban and rural areas. This will help them understand the difficulties we often face with people, environment, culture, geography, etc. ‘no expectem evri ting bae stret’ [Don’t expect everything to go right].” Man, Honiara
“They have their own charters, sometimes we might want to go another way but they don’t want to touch that. So sometimes there is some conflict there; some projects are not really what we would like to address – because the donors only want to do one component, and not another, because it is sensitive, or because they want quick results and to get out.” Government official, Honiara
“What changes have I noticed since independence? Whatever development you see here is due to individual struggles. No single aid program is sustainable. NGOs are created by donors and are comfortable with who they know. NGOs eat up the bulk of help intended for the communities. NGOs become international employers. They do their own thing in our province. Most projects have no impact. I want to say stop all aid except for education and health. If international assistance concentrates on quality education and health, the educated and healthy people will take care of themselves.” Government official, Auki, Malaita
“The most important impacts of aid people do not think about – they are not listed, not planned, they are remote, but these are the longest lasting. Often they are the opposite of the stated objectives. So remote, unintended, unexpected impacts are very often more important and more lasting and more dramatic than the short term intended, measured ones.” Aid consultant, Honiara
How widespread is our myopic focus on our intended results? A recent Netherlands Foreign Aid IOB study found unintended effects were an evaluator’s blindspot as across 664 evaluations over 20 years, “The ‘text miners’ found that only 1 in 6 IOB evaluation documents pay attention to unintended effects.” This dearth of attention to all the other things happening in projects led to 10 micro, macro, meso, and multiple level effects, from negative price effects such as food aid on local food producers or nationalist backlash to Afghan projects to positive effects (they found 40% of projects had this) such as a harbor built happening to expand beach tourism as well.
But if we don’t look for such effects, we don’t know the true impact of our aid programming. We also don’t honor the breadth of people’s lives rather than just as narrow ‘aid beneficiaries’ (ugh), not even honoring them with the terms’ participants’ much less ‘partners’ in their own development).
In our ex-post work, we find a wide array of ways ownership and implementation of activities is done after donors leave and without additional or with different resources, capacities, and partnerships. Taking emerging outcomes and impacts examples from a different Niger project, and one from Nepal:
1. Partnerships & Ownership: Half of the members of the all-women Village Banks reported helping one another deal with domestic disputes and violence. (Pact/Nepal)
2. Capacities: Trained local women charged rates to sell course materials onward (PACT/Nepal)
3. Ownership: Participants valued clinic-based birthing and sustained it by introducing locally-created social punishments and incentives (CRS/Niger)
4. Resources: New Ministry funding reallocated to sustain [health] investments, and private traders generated large crop purchases and contracts (CRS/Niger)
The assets and capacities we bring to help people and their country systems help only a sliver of their lives, and often in unexpected ways that sadly we aid donors and implementers don’t seem interested in. There are other puzzle pieces to add…
Let us not forget, as a Sustainable Development Goals Evaluation colleague said in 2017 on a call:
I am sure you, my dear colleagues, have reams of similar findings from your fieldwork. Please share yours!