Inter-American Development Bank (IDB) – where have your ex-post evaluations, and learning from them, gone?
A Linkedin colleague, Gillian Marcelle, Ph.D. recently asked me about ex-posts by the Inter-American Development Bank (IDB) as more Caribbean accelerators/incubators were planned without learning from previous identical tech investments. Here is what I found, and if anyone knows more, please contact me, as it is not reassuring. Also, some were internal ‘self-evaluations’, some were desk reviews, and only a few involved going to the field to ask aid recipients about what lasted, which is typical for multilaterals (ADB and the IBRD do the same). Given Valuing Voices’ focus on participant’s voices in results, there was an attempt to focus on those, but this report did not make it clear which were which so highlights are presented below.
In 2003 IDB created an ex-post policy, “Ex Post Policy (EPP) in October 2003, which mandated two new tasks to OVE: the review and validation of Project Completion Reports and the implementation of ex post project evaluations.”. These were under the Board’s request for “a commitment to a ‘managing for results’ business model.” 2004’s ex-posts were seen as “the first year of the implementation of the EPP, all 16 evaluations can be considered part of the pilot and the findings presented in this report refer to the entire set of ex post evaluations.” Further, “the general evaluative questions proposed by EPP are first “… the extent to which the development objectives of IDB-financed projects have been attained.” and second “… the efficiency with which those objectives have been attained.”
They spent over $300,000 unsuccessfully evaluating six of the projects. In part this was due to data quality. “six had an evaluation strategy identified in the approval stage, most had abandoned the strategy during execution prior to project closure and, with one exception which produced data that could be used to calculate a treatment effect, none had produced quality evaluative information…. No [Project Completion Report] PCR provided adequate information regarding the evolution of development outcomes expected from the project or an update with respect to the evaluation identified at the time of approval. For the other six, they found that the expected results did not match what would be the sustained results. Some were better than expected while more were worse. “A critical finding across all projects is the lack of correspondence between the reflexive estimates and the treatment effect estimates. In practically all cases, the estimates were different.” How sustainably “Improved” are “Lives” as IDB’s logo touts?
In 2004 IADB chose 16 projects to evaluate and dropped four for a variety of reasons. The remaining dozen projects ex-post evaluated were on land development, improving neighborhoods, and cash programming. There were data quality and comparability issues from the onset. In the land [tenure] ‘regularization’: “Six of the projects mention ex post evaluations in loan proposals, but none have been completed to date. OVE was successful in retrofitting a subset of outcomes expected for three projects: an attrition rate of 50%.” The neighborhood improvements had positive and negative results, with ‘retrofitting’ being needed regarding data. For the four evaluated, the overall conclusion was mixed. Both, that the projcts led to “greater coverage of certain public services.” and for two cases, “this impact was more pronounced for the poorest segments included in the treated population.” Nonetheless, much more was unachieved. “Beyond this, very little else can be said.The impact on the objectives related to human capital formation and income were not demonstrated. In the case of health interventions, perhaps the intervention type most directly linked to sanitation services; there has been no demonstrated link between the interventions and outcomes, even for the poorest segments of the beneficiary population. There was also no consistent evidence showing an increase in variables related to housing values.”
Regarding cash programming, there were individual evaluations that showed promise but only after statistical analysis of a control group, something which is sorely lacking in most foreign aid evaluations. An IDB project in Panama “shows that in some cases the reflexive evaluation, in fact, understated the true program treatment effect. The development outcome of this project was the reduction in poverty. A reflexive evaluation (the gross effects) of the incidence of poverty suggested that not only was the project unsuccessful but that it actually contributed to worsening poverty; the opposite of its intent. However, a treatment effect evaluation (the net effect) that compared “similar municipalities” shows that municipalities benefiting from FIS funds had a significant decline in poverty relative to comparable municipalities that did not receive FIS financing; the project had clear positive development outcomes”.
The IDB staff consulted in 2005 about the results “questioned whether the analysis of closed projects that were not required to include the necessary outcomes and data at the time of approval was a cost-effective use of Bank resources” which may be a reason why the Bank decided against doing more, in spite of many ex-post findings contradicting expected results. Astonishingly, since then, only one summary of a Jordanian ex-post in 2007 was found, but it is questionable that it is an actual ex-post closure. At a minimum, one would expect that the Bank would ensure data quality improved, and planned strategies would actually be done.
Finally, presumably a bank cares about Return on Investment. As a former investment banker, I would be concerned about the lack of learning, given the low cost of such learning versus the discrepancies found between expected and actual sustainability. Specifically, the extremely low cost of the six evaluations that were done ($113K each, more precisely a cost of .001-.21% of the program value), which is a pittance compared to the millions in loan values. Given that more were not done – or at least publically shared- in the 18 years since, sustainability-aware donors, beware.
Unlike this multilateral, I’ve been busy with two ex-post evaluations which I hope to share in the coming months… Let me know your thoughts!
or why sustainable results are so hard to come by….
A while ago, I reacted to a discussion among development aid/cooperation evaluators about why there are so few NGO evaluations available. It transpired that many people do not even know what they often look like, which is why I wrote a kind of Common Denominator Report: only about small evaluations, and self-explanatory in the sense that one understands why they are rarely published. The version below is slightly different from the original.
Most important elements of a standard evaluation report for NGOs and their donors; about twenty days of work; about 20,000 Euros budget (TVA included).
In reality, the work takes at least twice as much time as calculated and will still be incomplete/ quick, and dirty because it cannot decently be done within the proposed framework of conditions and answering all 87 questions or so that normally figure in the ToR.
The main issues in the project/ programme, the main findings, the main conclusions, and the main recommendations, presented in a positive and stimulating way (the standard request from the Comms and Fundraising departments) and pointing the way to the sunny uplands. This summary is written after a management response to the draft report has been ‘shared with you’. The management response normally says:
this is too superficial (even if you explain that it could not be done better, given the constraints);
this is incomplete (even if you didn’t receive the information you needed);
this is not what we asked (even if you had an agreement about the deliverables)
you have not understood us (even if your informants do not agree among themselves and contradict each other)
you have not used the right documents (even if this is what they gave you)
you have got the numbers wrong; the situation has changed in the meantime (even if they were in your docs);
your reasoning is wrong (meaning we don’t like it);
the respondents to the survey(s)/ the interviews were the wrong ones (even if the evaluand suggested them);
we have already detected these issues ourselves, so there is no need to put them in the report (meaning don’t be so negative).
Who the commissioning organisation is, what they do, who the evaluand is, what the main questions for the evaluators were, who got selected to do this work, and how they understood the questions and the work in general.
In the Terms of Reference for the evaluation, many commissioners already state how they want an evaluation done. This list is almost invariably forced on the evaluators, thereby reducing them from having independent status to being the ‘hired help’ from a Temp Agency:
briefings by Director and SMT [Senior Management Team] members for scoping and better understanding;
desk research leading to notes about facts/ salient issues/ questions for clarification;
survey(s) among a wider stakeholder population;
20-40 interviews with internal/ external stakeholders;
analysis of data/ information;
processing feedback on the draft report.
In the Terms of Reference, many commissioners already state which deliverable they want and in what form:
round table/ discussion of findings and conclusions;
presentation to/ discussion with selected stakeholders.
PROJECT/ PROGRAMME OVERVIEW
Many commissioners send evaluators enormous folders with countless documents, often amounting to over 3000 pages of uncurated text with often unclear status (re authors, purpose, date, audience) and more or less touching upon the facts the evaluators are on a mission to find. This happens even when the evaluators give them a short list with the most relevant docs (such as grant proposal/ project plan with budget, time and staff calculations, work plans, intermediate reports, intermediate assessments, and contact lists). Processing them leads to the following result:
According to one/ some of the many documents that were provided:
the organisation’s vision is that everybody should have everything freely and without effort;
the organisation’s mission is to work towards having part of everything to not everybody, in selected areas;
the project’s/ programme’s ToC indicates that if wishes were horses, poor men would ride;
the project’s/ programme’s duration was four/ five years;
the project’s/ programme’s goal/ aim/ objective was to provide selected parts of not everything to selected parts of not everybody, to make sure the competent authorities would support the cause and enshrine the provisions in law, The beneficiaries would enjoy the intended benefits, understand how to maintain them and teach others to get, enjoy and amplify them, that the media would report favourably on the efforts, in all countries/ regions/ cities/ villages concerned and that the project/ programme would be able to sustain itself and have a long afterlife;
the project’s/ programme’s instruments were fundraising and/ or service provision and/ or advocacy;
the project/ programme had some kind of work/ implementation plan.
This is where practice meets theory. It normally ends up in the report like this:
Due to a variety of causes:
unexpectedly slow administrative procedures;
funds being late in arriving;
bigger than expected pushback and/ or less cooperation than hoped for from authorities- competitors- other NGOs- local stakeholders;
sudden changes in project/ programme governance and/ or management;
incomplete and/ or incoherent project/ programme design;
incomplete planning of project/ programme activities;
social unrest and/ or armed conflicts;
The project/ programme had a late/ slow/ rocky start. Furthermore, the project/ programme was hampered by:
partial implementation because of a misunderstanding of the Theory of Change which few employees know about/ have seen/ understand, design and/ or planning flaws and/ or financing flaws and/ or moved goalposts and/ or mission drift and/ or personal preferences and/ or opportunism;
a limited mandate and insufficient authority for the project’s/ programme’s management;
high attrition among and/ or unavailability of key staff;
a lack of complementary advocacy and lobbying work;
patchy financial reporting and/ or divergent formats for reporting to different donors taking time and concentration away;
absent/ insufficient monitoring and documenting of progress;
little or no adjusting because of absent or ignored monitoring results/ rigid donor requirements;
limited possibilities of stakeholder engagement with birds/ rivers/ forests/ children/ rape survivors/ people in occupied territories/ murdered people/ people dependent on NGO jobs & cash etc;
internal tensions and conflicting interests;
neglected internal/ external communications;
un/ pleasant working culture/ lack of trust/ intimidation/ coercion/ culture of being nice and uncritical/ favouritism;
the inaccessibility of conflict areas;
Although these issues had already been flagged up in:
the evaluation of the project’s/ programme’s first phase;
the midterm review;
the project’s/ programme’s Steering Committee meetings;
the project’s/ programme’s Advisory Board meetings;
the project’s/ programme’s Management Team meetings;
Very little change seems to have been introduced by the project managers/ has been detected by the evaluators.
In terms of the OECD/ DAC criteria, the evaluators have found the following:
relevance – the idea is nice, but does it cut the mustard?/ others do this too/ better;
coherence – so so, see above;
efficiency – so so, see above;
effectiveness – so so, see above;
impact – we see a bit here and there, sometimes unexpected positive/ negative results too, but will the positives last? It is too soon to tell, but see above;
sustainability – unclear/ limited/ no plans so far.
If an organisation is (almost) the only one in its field, or if the cause is still a worthy cause, as evaluators you don’t want the painful parts of your assessments to reach adversaries. This also explains the vague language in many reports and why overall conclusions are often phrased as:
However, the obstacles mentioned above were cleverly navigated by the knowledgeable and committed project/ programme staff in such a way that in the end, the project/ programme can be said to have achieved its goal/ aim/ objective to a considerable extent.
Galileo: “Eppur si muove” = “And yet it moves”
Most NGO commissioners make drawing up a list of recommendations compulsory. Although there is a discussion within the evaluation community about evaluators’ competence to do precisely that, many issues found in this type of evaluation have organisational; not content; origins. The corresponding recommendations are rarely rocket science and could be formulated by most people with basic organisational insights or a bit of public service or governance experience. Where content is concerned, many evaluators are selected because of their thematic experience and expertise, so it is not necessarily wrong to make suggestions.
They often look like this:
Project/ programme governance
limit the number of different bodies and make remit/ decision making power explicit;
have real progress reports;
have real meetings with a real agenda, real documents, real minutes, real decisions, and real follow-up;
Project/ programme management
review and streamline/ rationalise structure to reflect strategy and work programme;
give project/ programme leaders real decision making and budgetary authority;
have real progress meetings with a real agenda, real minutes, real decisions, and real follow-up;
implement what you decide, but monitor if it makes sense;
consult staff on recommendations/ have learning sessions;
draft implementation plan for recommendations;
carry them out;
Processes and Procedures
get staff agreement on them;
commit them to paper;
stick to them – but not rigidly;
Obviously, if we don’t get organisational structure and functioning, programme or project design, implementation, monitoring, evaluation, and learning right, there is scant hope for the longer-term sustainability of the results that we should all be aiming for.
5 Ways to Foster Sustainability and Resilience to Climate Change
by Omar Abdou, M.A. and Jindra Cekan/ova, Ph.D
In the global climate change graphic, the blue are countries that contribute very few emissions to #climatechange. They are also those who suffer the most from climate change effects and who have most #globalaid projects which are most at risk of no longer being #sustained:
Omar Abdou, a food security and M&E expert from Niger, lives a vulnerability to climate change there is indisputable. Sahelien production systems suffer the full brunt of the negative effects. Indeed, the essential rural activities, namely agriculture, and livestock which occupy at least 80% of the population, are becoming more and more uncertain as they rely on two pillars: rainfall and the exploitation of natural resources (soil, vegetation, other water sources). But, in recent decades, we have witnessed a drastic reduction in rainfall. In pastoral zones, this decrease is reflected in the degradation of the vegetation cover and the deficits in livestock-vital fodder. For instance, the fodder assessment from 2000 to 2020 for the Tahoua Region has been in deficit for 15 years. More and more recurrent and frequent crises seriously affect the main productive capital of households, which is the herd.
Emergency sales of increasingly emaciated animals at a low price are growing to pay for consumer goods and to buy fodder to feed a few animals. These are kept for breeding after the crisis, but what used to be seasonal crisis (dry-season) has become chronic.
In agro-pastoral zones, agricultural production in the rainy season, which has changed its duration and strength, becomes uncertain to feed the household members. The harvest is often exhausted at most by the 6th month of the year for many households. Thus, there is no more food surplus to supply the area’s food needs where cereals are not grown. Suddenly, cereals become expensive and inaccessible for many families. Their coping strategies are diverse but not always effective. In pastoral settings, parents are forced to take their children out of school to go further south in search of food and pasture. Sometimes more sedentary agro-pastoralist farmers pledge or sell their land to pay for food. Those left behind who have nothing to sell, are forced to cut trees and sell the wood for charcoal, further degrading the environment. Unfortunately, Niger’s neighbors are now mostly at war, so these young men run the risk of being recruited by armed groups or are killed. In these conditions, there is serious instability where the means of production (natural and human) are sold for survival and overexploited natural resources decrease. Increasingly, it is practically impossible to talk about sustainability. It is, therefore, necessary to think of consistent and innovative projects to enable households to produce sustainably while preserving the ecosystem for future generations. Outlets such as migration of the able-bodied workers to neighboring countries to provide labor in the lean season work now but are unsustainable.
Jindra notes that 31 Oct the UN Climate Change Conference (COP26) begins. Climatescience documents life-threatening effects such as Omar and billions of others are already experiencing around the ‘developing’ world. The IPCC Working Group report shows that “the world will probably reach or exceed 1.5 degrees C (2.7 degrees F) of warming within just the next two decades. Whether we limit warming to this level and prevent the most severe climate impacts depends on actions taken this decade. Only with ambitious emissions cuts can the world keep global temperature rise to 1.5 degrees C, the limit scientists say is necessary for preventing the worst climate impacts. Under a high-emissions scenario, the IPCC finds the world may warm by 4.4 degrees C by 2100 — with catastrophic results.” Erratic rainfall, greater heatwaves will affect 70% of all farmers and herders who rely on rain for their production and a new report on Adaptation to climate across West African shows aridity increasing, which must inform global development.
What can we do:
1- LISTEN AND ACT GLOBALLY: There are excellent resources to track demands that the Global North divest and stop subsidizing fossil fuels. For instance, “In 2017/18, the G7 collectively spent more on fossil fuel finance than on climate finance, according to analysis by Oil Change International. They allocated nearly $40bn that year.” Hypocrisy abounds as this Climate Change News article goes on to note that “Since the Paris Agreement was signed, G20 countries’ export credit agencies have provided 14 times as much support for fossil fuels as clean energy “to $64 billion a year, and investors still prioritize profit over planet & people.
3- DESIGN CLIMATE SMARTLY: Global Development needs to do much more climate-smart programming as 50 years of investments in foreign aid to ‘developing’ countries has already started to be threatened. What can be done? Fromsequestering the carbon content of soil via farming to fostering climate-smart agriculture, which, according to the UN’s FAO means “agriculture that sustainably increases productivity, enhances resilience (adaptation), reduces/removes GHGs (mitigation) where possible, and enhances achievement of national food security and development goals”. Importantly this involves productivity plus adaptation and mitigation against changes wrought by climate change.
4- MEASURE SUSTAINABILITY: Exciting measurements are beginning!Scotland is valuing its nature, and there are calls for The Rights Of Nature In Evaluation Of Environmental Sustainability. Jindra is consulting to a wonderful team at the Adaptation Fund. We are evaluating what could be sustained, and what withstood climatic changes is ex-post project sustainability and resilience evaluation. The Adaptation Fund on evaluating projects’ longer-term (3-5 year ex-post project closure) sustainability and, importantly their resilience to climate change. We are piloting these rare kinds of evaluations in six countries, starting with Samoa and Ecuador in 2021. As the Fund has funded over 100 projects in 100 countries, with almost 30 million participants, we have a rich array to choose from. Such a learning opportunity about what lasts and has withstood disturbances is vital. As the Fund notes: “Climate change is predicted to greatly affect the poorest people in the world, who are often hardest hit by weather catastrophes, desertification, and rising sea levels, but who have contributed the least to the problem of global warming. In some parts of the world, climate change has already contributed to worsening food security, reduced the predictable availability of freshwater, and exacerbated the spread of disease and other threats to human health. Helping the most vulnerable countries and communities is an increasing challenge and imperativefor the international community, especially because climate adaptation requires significant resources beyond what is already needed to achieve international development objectives.”
5- TAKE PERSONAL RESPONSIBILITY: For 8 years, Valuing Voices has pushed back to evaluate the ‘Sustainable Development” we have promised for 50 years. We must be accountable primarily to our participants and partners, rather than donors, and only by listening to these country nationals, we can learn what was sustained not, why or why not, and what emerged from their efforts. In 2019, we wrote all of us are accountable to improve the SDGs. It is wonderful to have climate change leading organizations invest in ex-post evaluation that adds resilience, prioritizing organizational learning and accountability. We all must increase our awareness that curb our consumption, for the sake of Niger, small island states, and many other parts of our planet can no longer stand our business-as-usual. Limiting climate change begins with us, from changing our food consumption to consumer choices to political advocacy to seeing the Earth as indivisible from us and us from the Earth. Let us start to invest in global sustainability today.
Do you see any bright spots on the horizon? Do share!
Ex-post Eval Week: Exiting For Sustainability by Jindra Cekan
Reblogged from AEA: https://aea365.org/blog/ex-post-eval-week-exiting-for-sustainability-by-jindra-cekan-2/ January 22, 2021
Hello. My name is Jindra Cekan, and I am the Founder and Catalyst of Valuing Voices at Cekan Consulting LLC. Our research, evaluation and advocacy network have been working on post-project (ex-post) evaluations since 2013. I have loved giraffes for decades and fund conservation efforts (see pix).
Our planet is in trouble as are millions of species, including these twiga giraffes and billions of homo-sapiens. Yet in global development we evaluate projects based on their sectoral, e.g. economic, social, educational, human rights etc., results, with barely a glance at the natural systems on which they rest. IDEAS Prague featured Andy Rowe and Michael Quinn Patton who showed that I too have been blind to this aspect of sustainability.
I have argued ad nauseum that the OECD’s definition of projected sustainability and impact don’t give a hoot about sustaining lives and livelihoods.. If we did, we would not just claim we do ‘sustainable development’ and invest in ‘Sustainable Development Goals’ but go about proving how well, for how long, by whom, after closeout.
After hearing Rowe, I added to my Sustained Exit Checklists new elements about how we must evaluate Risks to Sustainability and Resilience to Shocks that included the natural environment. I added Adaptation to Implementation based on feedback on how much implementation would need to change based in part on climatic changes.
Yet new evaluation thinking by Rowe, Michael Quinn Patton, Astrid Brouselle/ Jim McDavid take us a quantum leap beyond. We must ask how can any intervention be sustained without evaluating the context in which it operates. Is it resilient to environmental threats? Can participants adapt to shocks,? Have we assessed and mitigated the environmental impacts of our interventions? As Professor Brouselle writes, “changing our way of thinking about interventions when designing and evaluating them…. away from our many exploitation systems that lead to exhaustion of resources and extermination of many species.”
This 2020 new thinking includes ascertaining:
(Andy Rowe) Ecosystems of biotic natural capital and abiotic natural capital (from trees to minerals) with effects on health, education, public safety/ climate risk and community development
(Astrid Brouselle and Jim McDavid) Human systems that affect our interventions, including: Power relations, prosperity, equity and we need to make trade-offs between environment and development goals clear.
We have miles to go of systems and values to change. Please read this and let’s start sustaining NOW.
Rowe, A. (2019). Sustainability-read evaluation: A call to action. In G. Jules (Ed.) Evaluating Sustainability: Evaluative Support for Managing Processes in the Public Interest. New Directions for Evaluation, 162, 29-48.
This week, AEA365 is celebrating Ex-post Eval Week during which blog authors share lessons from project exits and ex-post evaluations. Am grateful to the American Evaluation Association that we could share these resources….
Reblog Ex-post Eval Week: Evaluating Peace for Sustainability by Peter Kimeu Ngui
Reblogged from https://aea365.org/blog/ex-post-eval-week-evaluating-peace-for-sustainability-by-peter-kimeu-ngui/ January 20, 2021
Peace is with you. My name is Peter Kimeu Ngui, the Founder and CEO of Decent Living Institute of Organic Farming with extensive experience on reflective peace and partnership building in Africa.
Is there anything like a ‘just world’? Can peace be ‘sustainable’? These two closely related questions must be in mind of evaluators of peace for sustainability in a community long after the funding expires. According to Catholic Relief Services’ Justice Lens, three key conditions that must exist in the vision of a just world and used by all sustainable peace evaluators.:
(1) sustainable right relationship for all humanity and nature
(2) the transformation of injustices in society, structures and policies with
(3) fairness of rights and responsibilities for sustained peace to all creation
The process of reflective peace building evaluation, including participatory creation of sustainability indicators demands a collaborative effort by all the stakeholders. This includes victims, perpetrators, peace builders/ advocacy agents who are necessary to establish and maintain a just environment.
Step one of the process involves reflecting on eight keys principles drawn from Pope Francis’ letter that awaken the conscience and human rights standards which fit into an acronym (CORDSSSS) for easy remembrance; Common Good, Option for the Poor, Rights and Responsibilities, Dignity and Equality, Social nature, Subsidiarity, Stewardship and Solidarity. Each team reflects on:
the definition of each principle
discussing what their world would be like if the principle was fully lived
what would be sustainable peace indicators (e.g. Increased peaceful resolutions administered by the community; peaceful co-existence by people in violent conflicts; reduced rape)
what needs to be different in the world (location) they live in?
Step two of the process is scenario building using the Appreciative Inquiry method that engages the participants in the evaluation. This involves finding out what:
sustainable values, customs, and traditions
policies and structures/ institutions exist in the community that promote/ advocate, promote, establish and maintain, monitor, and evaluate for sustainable peace building
the participants are led to dreaming of a just environment in their community and what needs be improved and put in place for this scenario to be realized.
The evaluation teams are always and immediately energized to pursue the best scenario for peacebuilding, using the resources locally available. The reflective peacebuilding process engages participants on real issues that directly affects the community, that the community is fears, worries, concerned with and they are immediately empowered with knowledge, using familiar traditional approach to resolve the violent conflict, and they self-discover that power to transform is with those engaged in the conflict.
The possibility of a just world becomes a reality amassing greater numbers of individuals to work for peace and work through these processes.
Lessons learned are self-created from local case study evaluation processes and appeal greatly to most of the community members
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).