The Contentious Power of Evaluations, Guest Blog by Hanneke de Bode

by | Jun 27, 2022 | Aid effectiveness, Evaluation, Evaluation Findings, Learning, NGOs, OECD, OECD DAC, Sustainability, Sustainable development, TOR

THE CONTENTIOUS POWER OF EVALUATIONS

or why sustainable results are so hard to come by….

 

A while ago, I reacted to a discussion among development aid/cooperation evaluators about why there are so few NGO evaluations available. It transpired that many people do not even know what they often look like, which is why I wrote a kind of Common Denominator Report: only about small evaluations, and self-explanatory in the sense that one understands why they are rarely published. The version below is slightly different from the original.

 

Most important elements of a standard evaluation report for NGOs and their donors; about twenty days of work; about 20,000 Euros budget (TVA included).

 

In reality, the work takes at least twice as much time as calculated and will still be incomplete/ quick, and dirty because it cannot decently be done within the proposed framework of conditions and answering all 87 questions or so that normally figure in the ToR.

 

EXECUTIVE SUMMARY

The main issues in the project/ programme, the main findings, the main conclusions, and the main recommendations, presented in a positive and stimulating way (the standard request from the Comms and Fundraising departments) and pointing the way to the sunny uplands. This summary is written after a management response to the draft report has been ‘shared with you’. The management response normally says:

  • this is too superficial (even if you explain that it could not be done better, given the constraints);
  • this is incomplete (even if you didn’t receive the information you needed);
  • this is not what we asked (even if you had an agreement about the deliverables)
  • you have not understood us (even if your informants do not agree among themselves and contradict each other)
  • you have not used the right documents (even if this is what they gave you)
  • you have got the numbers wrong; the situation has changed in the meantime (even if they were in your docs);
  • your reasoning is wrong (meaning we don’t like it);
  • the respondents to the survey(s)/ the interviews were the wrong ones (even if the evaluand suggested them);
  • we have already detected these issues ourselves, so there is no need to put them in the report (meaning don’t be so negative).

 

BACKGROUND

Who the commissioning organisation is, what they do, who the evaluand is, what the main questions for the evaluators were, who got selected to do this work, and how they understood the questions and the work in general.

 

METHODOLOGY

In the Terms of Reference for the evaluation, many commissioners already state how they want an evaluation done. This list is almost invariably forced on the evaluators, thereby reducing them from having independent status to being the ‘hired help’ from a Temp Agency:

  • briefings by Director and SMT [Senior Management Team] members for scoping and better understanding;
  • desk research leading to notes about facts/ salient issues/ questions for clarification;
  • survey(s) among a wider stakeholder population;
  • 20-40 interviews with internal/ external stakeholders;
  • analysis of data/ information;
  • recommendations;
  • processing feedback on the draft report.

 

DELIVERABLES

In the Terms of Reference, many commissioners already state which deliverable they want and in what form:

  • survey(s);
  • interviews;
  • round table/ discussion of findings and conclusions;
  • draft report;
  • final report;
  • presentation to/ discussion with selected stakeholders.

 

PROJECT/ PROGRAMME OVERVIEW

 

Many commissioners send evaluators enormous folders with countless documents, often amounting to over 3000 pages of uncurated text with often unclear status (re authors, purpose, date, audience) and more or less touching upon the facts the evaluators are on a mission to find. This happens even when the evaluators give them a short list with the most relevant docs (such as grant proposal/ project plan with budget, time and staff calculations, work plans, intermediate reports, intermediate assessments, and contact lists). Processing them leads to the following result:

 

According to one/ some of the many documents that were provided:

  • the organisation’s vision is that everybody should have everything freely and without effort;
  • the organisation’s mission is to work towards having part of everything to not everybody, in selected areas;
  • the project’s/ programme’s ToC indicates that if wishes were horses, poor men would ride;
  • the project’s/ programme’s duration was four/ five years;
  • the project’s/ programme’s goal/ aim/ objective was to provide selected parts of not everything to selected parts of not everybody, to make sure the competent authorities would support the cause and enshrine the provisions in law, The beneficiaries would enjoy the intended benefits, understand how to maintain them and teach others to get, enjoy and amplify them, that the media would report favourably on the efforts, in all countries/ regions/ cities/ villages concerned and that the project/ programme would be able to sustain itself and have a long afterlife;
  • the project’s/ programme’s instruments were fundraising and/ or service provision and/ or advocacy;
  • the project/ programme had some kind of work/ implementation plan.

 

FINDINGS/ ANALYSIS

 

This is where practice meets theory. It normally ends up in the report like this:

 

Due to a variety of causes:

  • unexpectedly slow administrative procedures;
  • funds being late in arriving;
  • bigger than expected pushback and/ or less cooperation than hoped for from authorities- competitors- other NGOs- local stakeholders;
  • sudden changes in project/ programme governance and/ or management;
  • incomplete and/ or incoherent project/ programme design;
  • incomplete planning of project/ programme activities;
  • social unrest and/ or armed conflicts;
  • Covid;

 

The project/ programme had a late/ slow/ rocky start. Furthermore, the project/ programme was hampered by:

  • partial implementation because of a misunderstanding of the Theory of Change which few employees know about/ have seen/ understand, design and/ or planning flaws and/ or financing flaws and/ or moved goalposts and/ or mission drift and/ or personal preferences and/ or opportunism;
  • a limited mandate and insufficient authority for the project’s/ programme’s management;
  • high attrition among and/ or unavailability of key staff;
  • a lack of complementary advocacy and lobbying work;
  • patchy financial reporting and/ or divergent formats for reporting to different donors taking time and concentration away;
  • absent/ insufficient monitoring and documenting of progress;
  • little or no adjusting because of absent or ignored monitoring results/ rigid donor requirements;
  • limited possibilities of stakeholder engagement with birds/ rivers/ forests/ children/ rape survivors/ people in occupied territories/ murdered people/ people dependent on NGO jobs & cash etc;
  • internal tensions and conflicting interests;
  • neglected internal/ external communications;
  • un/ pleasant working culture/ lack of trust/ intimidation/ coercion/ culture of being nice and uncritical/ favouritism;
  • the inaccessibility of conflict areas;

 

Although these issues had already been flagged up in:

  • the evaluation of the project’s/ programme’s first phase;
  • the midterm review;
  • the project’s/ programme’s Steering Committee meetings;
  • the project’s/ programme’s Advisory Board meetings;
  • the project’s/ programme’s Management Team meetings;

 

Very little change seems to have been introduced by the project managers/ has been detected by the evaluators.

 

In terms of the OECD/ DAC criteria, the evaluators have found the following:

  • relevance – the idea is nice, but does it cut the mustard?/ others do this too/ better;
  • coherence – so so, see above;
  • efficiency – so so, see above;
  • effectiveness – so so, see above;
  • impact – we see a bit here and there, sometimes unexpected positive/ negative results too, but will the positives last? It is too soon to tell, but see above;
  • sustainability – unclear/ limited/ no plans so far.

 

OVERALL CONCLUSION

 

If an organisation is (almost) the only one in its field, or if the cause is still a worthy cause, as evaluators you don’t want the painful parts of your assessments to reach adversaries. This also explains the vague language in many reports and why overall conclusions are often phrased as:

 

However, the obstacles mentioned above were cleverly navigated by the knowledgeable and committed project/ programme staff in such a way that in the end, the project/ programme can be said to have achieved its goal/ aim/ objective to a considerable extent.

 

Galileo: “Eppur si muove” = “And yet it moves”

 

 

RECOMMENDATIONS

 

Most NGO commissioners make drawing up a list of recommendations compulsory. Although there is a discussion within the evaluation community about evaluators’ competence to do precisely that, many issues found in this type of evaluation have organisational; not content; origins. The corresponding recommendations are rarely rocket science and could be formulated by most people with basic organisational insights or a bit of public service or governance experience. Where content is concerned, many evaluators are selected because of their thematic experience and expertise, so it is not necessarily wrong to make suggestions.

They often look like this:

 

Project/ programme governance

  • limit the number of different bodies and make remit/ decision making power explicit;
  • have real progress reports;
  • have real meetings with a real agenda, real documents, real minutes, real decisions, and real follow-up;
  • adjust;

Project/ programme management

  • review and streamline/ rationalise structure to reflect strategy and work programme;
  • give project/ programme leaders real decision making and budgetary authority;
  • have real progress meetings with a real agenda, real minutes, real decisions, and real follow-up;
  • implement what you decide, but monitor if it makes sense;
  • adjust;

Organisational management

  • consult staff on recommendations/ have learning sessions;
  • draft implementation plan for recommendations;
  • carry them out;

Processes and Procedures

  • get staff agreement on them;
  • commit them to paper;
  • stick to them – but not rigidly;

Obviously, if we don’t get organisational structure and functioning, programme or project design, implementation, monitoring, evaluation, and learning right, there is scant hope for the longer-term sustainability of the results that we should all be aiming for.

Valuing Voices Better Ex-Post Evaluation White Paper: Nordics & the Netherlands

by | Jun 29, 2021 | ex-post evaluation, Norway, OECD, Project cycle, Sustainable development

Donors often talk about #SustainableDevelopment, and we know they try. But do they return #expost to #evaluate it? Here are lessons from #Nordic (#Norway, #Finland, #Sweden & #Denmark) and the #Netherlands in our #ValuingVoices #WhitePapers. Let us know what you think… Thanks, Jindra & Preston Stewart..

The document(s) cover:
1) the search process,
2) how ex-post evaluations are defined and categorized,
3) what was done well by each country’s ex-posts,
4) sustainability-related findings and lessons, and
5) what M&E experts in each country can improve on ex-post evaluation practices.
6) Case studies of the Netherlands, Norway, and Finland
One big finding is that there were only 32 evaluations that seemed to be ex-post project, and only 16 of them actually were at least 2 years after project closure.

Executive Summary/ Abstract of the set of White Papers

Ex-post completion or ex-post project evaluations are surprisingly rare given widespread commitments to ‘sustainable development’. Given the well-known commitment of the Nordic countries of Norway, Finland, Sweden, and Denmark as well as the Netherlands, Valuing Voices, a consulting firm specializing in ex-post project evaluation, chose these countries as case studies. We hired an intern from Harvard College, who searched the national databases (in English only) and presented these results. The findings were surprising. There were far fewer ex-posts in total than expected. Those that were included a surprising number that was co-evaluated alongside final evaluations, although final evaluation illuminates relevance, effectiveness, efficiency (and the new OECD criteria of coherence), while ex-post project evaluation illuminates sustainability and impact two or more years after closure.  There were lessons for Ministries about the searchability as well as the quality (and in the case of Denmark and in part, Sweden), the dearth of ex-posts at all. Overall, the recommendations for the five countries researched for this paper include better and more standardized definitions of ex-post project, greater absolute numbers done, transparent sharing of those done in publicly searchable manners,  methodologically clear comparisons (baseline and midterm), and clarity in differentiating different evaluations.

 

There are caveats to these papers. This research was privately by Valuing Voices so the samples featured from the five countries were limited to public sources about ex-post and ex-post evaluations.  While we reached out to Ministry evaluative staff in all countries, only two made themselves available for consultation (the Netherlands and Finland) and provided a handful of additional ex-post evaluations.  This paper focuses on what such research yielded, not definitive findings of programs or multi-year country strategies that are funded for 20-30 years continuously, nor projects funded by country-level embassies which did not feature on the Ministry site. We focus on project bilateral project evaluations, not multilateral funding of sectors. We also presented these findings in a 2021 webinar, during which we received input that Sweden’s EBA has a (non-project) portfolio of ‘country evaluations’ which looked back over 10 or even 20-year time horizons.. Still, we found very few ex-post evaluations at EBA (see cite in Search, below).  No input was received by Norway or Denmark.

 

There are four papers in this series combined in this White Paper. First, the search across databases of all five Nordics (Norway, Finland, Sweden, and Denmark) and the Netherlands (pages 1-21). This is followed by country-specific, detailed, papers for those with ex-posts: The Netherlands (pages 22-37), Norway (pages 38-47), and Finland (pages 48-59).  We look forward to your feedback. Jindra@ValuingVoices.com for the Valuing Voices team.

Let us know what you think… Thanks, Jindra & Preston Stewart

“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School

by | Apr 9, 2021 | business, environment, ESG, ex-post evaluation, foreign aid, Impact, impact investing, OECD, Sustainable development

“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School

Recently I’ve had conversations where I had to define which sustainability we were talking about. Was it:

  1. ex-post-project sustainability of outcomes and impacts,
  2. environmental sustainability, or
  3. business sustainability?

 

Since I spend most of my time evaluating the ex-post sustained and emerging impacts of foreign aid projects years after projects close, or at least advocate for it, let’s start there.

  1. The Organisation for Economic Co-operation and Development (OECD) is a “forum and knowledge hub for data and analysis, exchange of experiences, best-practice sharing, and advice on public policies and international standard-setting.” Regarding evaluation specifically, the OECD has “established common definitions for six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact, and sustainability – to support consistent, high-quality evaluation”. Focusing on long-term sustainability, their evaluation guidance is:

 

 

Source: OECD, Better Criteria for Better Evaluation, 2019

 

The good news is that in this recent publication on Applying Evaluation Criteria Thoughtfully (2021), OECD keeps the updated definition but inches towards recommending actual ex-post project sustainability evaluation, rather than just projected (and assumed “likely to continue” sustainability). For this, “likely” is the most significant reason evaluators for donors and implementers have assumed, rather than evaluated, sustainability for decades. Further, positive, ‘sustained’ trajectories are also assumed at close-out/ exit, but rarely tested ex-post.

The OECD criteria give not evaluating it as an option. I far prefer “net benefits of the intervention continue” as it is a marching order: Prove that results were sustained. In this evolution, this 2021 report states, “After the completion of the intervention, and evaluation of sustainability would look at whether or not the benefits did continue, this time drawing on data and evidence from the intervention’s actual achieved benefits.”

 

OECD even goes on to recommend implementing and monitoring for sustainability. The new piece de resistance is: “Sustainability should be considered at each point of the results chain and the project cycle of an intervention”:

  1. “The sustainability of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in the broader context of the intervention…. as well as whether there was willingness and capacity to sustain financing (resources) at the end of the intervention
  2. For example, an evaluation could assess whether an intervention considered partner capacities
  3. Built ownership at the beginning of the implementation period…. And
  4. In general, evaluators can examine the conditions for sustainability that were or were not created in the design of the intervention and by the intervention activities and whether there was adaptation where required.”

Yes, Valuing Voices highlighted this at the American Evaluation Association presentation “Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation” in 2016, and we have developed this into 2020’s Exiting for Sustainability trainings and checklists. Wonderful to see implementing for sustainability in guidance by the OECD! Many of the elements his research cites overlap with our Exiting for Sustainability checklislts which includes ownership, resources, capacities, partnerships, and implementation/ M&E process.

 

Moreover, while the 2019 OECD report mentioned resilience in passing, related to sustainability, “encourages analysis of potential trade-offs, and of the resilience of capacities/ systems underlying the continuation of benefits”. Such resilience and continuation of benefits evaluation involve examining huge systems (the financial, economic, social, environmental, and institutional capacities) that projects and programs are implemented within, whose stability is needed to sustain net benefits over time. Yes, for ex-post sustainability questions for evaluators to consider should include: “To what extent did the intervention contribute to strengthening the resilience of particularly disadvantaged or vulnerable groups” on which the sustained impacts of so much of our “Leave No One Behind” myth of Sustainable Development rely.

 

However, OECD makes suggestions to evaluate even broader, overwhelming what is feasible: “…this involves analyses of resilience, risks, and potential trade-offs.” Whose? All stakeholders, from participants to local partners and national and international implementers, and international donors? How far back and how far forward? What a huge undertaking. Further, the OECD points evaluators to define resilience, but as I learned in my Famine Early Warning System research and a current ex-post evaluation process for the Adaptation Fund, that involves creating evaluable boundaries by determining resilient to what kinds of shocks? Vital questions current industry monitoring and evaluation budgets for all evaluations, much less (too-rare) ex-post project evaluations, are insufficient for as they hover around 3-5% of total costs.

  1. Slight progress at OECD is being made by acknowledging environmental sustainability first brought up by the Brundtland Report, “Our Common Future” back in 1987. This linchpin report highlighted that “critical global environmental problems were primarily the result of the enormous poverty of the South and the non-sustainable patterns of consumption and production in the North. It called for a strategy that united development and the environment – described by the now-common term “sustainable development”… that meets the needs of the present without compromising the ability of future generations to meet their own needs.”

 

While an OECD brief in 2008 considers the environmental aspects of our thinking about sustainability, it argues that sustainability primarily about “using economic development to foster a fairer society while respecting ecosystems and natural resources.” The 2021 Applying Evaluation Criteria Thoughtfully rather unhelpfully mostly ignores the environment’s role in sustainability: “Confusion can arise between sustainability in the sense of the continuation of results, and environmental sustainability or the use of resources for future generations. While environmental sustainability is a concern (and maybe examined under several criteria, including relevance, coherence, impact, and sustainability), the primary meaning of the criteria is not about environmental sustainability as such; when describing sustainability, evaluators should be clear on how they are interpreting the criterion.” Given rapid climate change, I would argue that any sustained and emerging outcomes and impacts of projects that does not include an evaluation of the environmental context will fail to foster sustained resilience. Yet donors’ fixed funding timeframes that set completion to disbursement without evaluating sustainability or resilience continue to be huge barriers.

 

In 2022 a new resource on Sustainability from the media perspective questions the OECD. Sustainability: Going Beyond the Buzzword groups sustainability described in various media as: 1) Economic viability, 2) Social Sustainability, 3) Institutional Sustainability and 4) technical sustainability. But we’ve learned in 2022 from consulting work that the technology piece mentioned in this Buzzwords study regarding ‘baking in’ sustainability via the infrastructure/ assets created (The Adaptation Fund’s ex-post and resilience evaluations) and incorporating INGOs’ considerations of buying-local, seeing what local spare parts and technicians will be available in the future is a great added consideration. So too is promoting locally-led development through the very participatory way projects are implemented, with locals leading. Among the best resources is CDA/ Peace Direct/ Search for Common Ground’s Stopping as Success studies for USAID.

 

  1. Finally, business sustainability brings together these impacts on communities and society along with impacts on the environment. These are called ESG (Environmental, Social, and Governance) criteria. A Harvard Business School brief defines sustainability as “doing business without negatively impacting the environment, community, or society as a whole. “Where applied well, the aspiration is that “beyond helping curb global challenges, sustainability can drive business success.” While Harvard Business Review highlights “What Works’ in Calculating the Value of Impact Investing, they are, like almost all of global development ‘while- we-are-there’-measures. There is one mention of ‘terminal value’, 5 years after close of ownership, and they estimate social return on investments. This is a good, step, but as insufficient as foreign aid – for these are projected, not actual results.

At Valuing Voices, we have found hopeful examples such as IKEA as well as where ‘impact investing’ hype does not match the claims. Nonetheless, increasingly businesses are trying to consider circular economy systemic principles of “economic development designed to benefit businesses, society, and the environment.” This is regenerative, aims to decouple growth from the consumption of finite resources, not generate excess waste that cannot be reused and actuals seem to be measured at least during investments. As Harvard notes, “this leads investors to look at factors such as a company’s carbon footprint, water usage (both Environment), community development efforts (Social), and board diversity (Governance).” We encourage them to measure long-term/ longitudinally. A current Harvard Business Review sobering article on the ineffectiveness so far of measuring environmental sustainability and ESG. “…reporting is not a proxy for progress. Measurement is often nonstandard, incomplete, imprecise, and misleading. And headlines touting new milestones in disclosure and socially responsible investment are often just fanciful ‘greenwishing’”.

Australia’s RMIT defines business sustainability as comprising 4 pillars: Human, Social, Economic, and Environmental which combines a) “Human sustainability focuses on the importance of anyone directly or indirectly involved in the making of products, or provision of services or broader stakeholders;… b) Social sustainability focuses on maintaining and improving social quality with concepts such as cohesion, reciprocity and honesty and the importance of relationships amongst people;… c) Economic sustainability aims to improve the standard of living [and] the efficient use of assets to maintain company profitability over time;… d) Environmental sustainability places emphasis on how business can achieve positive economic outcomes without doing any harm, in the short- or long-term, to the environment.” But how well measured?!

Would ESG success be sustained over the long-term rather than short-term shareholder profit cycles? Will the OECD start to recommend extensive ex-post evaluation? Will they develop guidance to incorporate environmental concerns in evaluation for our common good? I do not yet know, but I implore these silos to start talking. No time to waste!

As my colleague and  collaborator Susan Legro commented, we need to:

1) Continue to seek clarity and specificity in the terminology that we use, ensuring that it is clear to all stakeholders and beneficiaries; and

2) Find ways to study projects and initiatives over the longer term, which is the only way to study the designation of “sustainable” for any initiatives seeking that label.

3) I’ll add spread the word, as I just found out Harvard’s Extension School course on Sustainability features this blog in its curriculum 🙂

What are your thoughts?

 

Sustaining “Sustainable Development”

by | Apr 20, 2020 | Aid effectiveness, climate change, environment, ex-post evaluation, Impact, international development, JICA, OECD, Participants, post project evaluation, Sustainability, Sustainable development, USAID, Vietnam, water/ sanitation

 

Sustaining “Sustainable Development”?

 

As a global development industry, we have almost no evidence of how (un)sustained the outcomes or impacts of 99% of our projects because we have never returned to evaluate them. But from early indications based on the ex-posts, we have evaluated 2-20 years after donor departure it is, learning from what was and was not sustained is vital before replication and assuming sustainability. Most results taper off quite quickly, showing 20-80% decreases as early as two years post-closure and donor exit. A few cases of good news also appear, but more trajectories falter and fail than rise or remain. Sustainability, then, is not a yes-no answer, but a how much, yet too few ask… hence if they were, resilient, they are less so, or even not at all, now.

 

At Valuing Voices we focus on the sustainability of projects after external support ends. Still, those projects are also dependent on the viability of the environment in which they are based. As Andy Rowe, an evaluator on the GEF’s Adaptation Fund board, noted at IDEAS’ Conference in Prague late 2019 [1], a need for sustainability-ready evaluation to help us know how viable the resources are on which so many of our projects rest [2]. He states, “the evaluation we have today treats human and natural systems as unconnected and rarely considers the natural system”. He goes on to differentiate between biotic natural capital  (air, water, plants, and trees) and abiotic natural capital sources (fossil fuels, minerals, and metals, wind, and solar).

 

How much are projects designed assuming those resources are and will remain plentiful? How often do we evaluate how much our projects drain or rely on these environmental elements? Many projects are required to do environmental compliance and safeguarding against damage at project onset [3]. Others, such as agriculture and natural resource management or water/ sanitation, often focus on improving the environment on which those activities rely, e.g., improving soil or terrain (e.g., terraces, zais), planting seedlings, and improving access to potable water for humans and animals. Still, many projects ‘assume’ inputs like rainfall, tree cover, solar power, or do not consider the sustainability of natural resources for the communities in which they intervene. Examples are both those that rely on natural systems as well as those supposedly beyond them, e.g., enterprise development, education, safety nets, etc. Yet many enterprises, schools, safety nets do rely on a. viable environment in which their participants trade, learn, and live, and all are subject to the growing climate change disruptions. 

 

Why is this urgent? The OECD/DAC reminds us that “Natural assets represent, on average 26% of the wealth of developing countries compared to 2% in OECD economies” [4]. Unless we protect them and address the demand for natural resources, demand will far outstrip supply. “By 2030, an additional 1 billion people are expected to live in severely water-stressed areas, and global terrestrial biodiversity is expected to decline an additional 10%, leading to a loss of essential ecosystem services. By 2050, growing levels of dangerous air emissions from transport and industry will increase the global number of premature deaths linked to airborne particulate matter to 3.6 million people a year, more than doubling today’s levels. Failure to act could also lead to a 50% increase in global greenhouse gas emissions by 2050, and global mean temperature increases of 3-6°C by the end of the century, in turn contributing to more severe and sometimes more frequent natural disasters… [so] reconciling development with environmental protection and sustainable resource management is broadly agreed as a central concern for the post-2015 development agenda.”

 

When we return to projects that are a mix of behavior change and environment, we find a wide range of results:

  • Some projects, such as JICA Vietnam’s water supply and irrigation infrastructure reached 80% of the final results two years later [5]. And while the pilot projects were worse off (as low as 28% of irrigated hectares), longer-standing projects sustained as much as 72% of final results. While such agricultural development assumes continued water supply and access, does it evaluate it? No.
  • Some can define what ex-post lessons are more narrowly as functioning mechanisms: New ex-posts of water/ sanitation showed better – but still mixed results, such as USAID Senegal’s [6]. “While a majority (63 percent) of the water points remained functional, the performance varied significantly based on the technology used. Of the different technologies, the Erobon rope pumps performed poorly (27 percent functional), while the India Mark (74 percent functional) and mechanized pumps (70 percent functional) performed the best.”
  • Some projects that include environmental considerations illustrate our point by only focusing on behavior change as this sanitation/ hygiene ex-post from Madagascar did, where results fell off precipitously three years ex-post but without considering water supply or quality much [7]. 

[7]

  • There can be useful learning when one combines an evaluation of both types of sustainability (ex-post and environmental). A JICA irrigation project in Cambodia shows that when irrigation canals were mostly sustained over the five-years ex-post, they could serve increasing needs for land coverage and rice production [7]. The area of irrigated fields at the national level in 2010 reached the target, and the irrigated field area has since continued to increase in most areas. Even the largest drop [in area irrigated] post-closure was only 11%. They reported that the unit yield of rice at the end-line survey in 2012 at 11 sites was 3.24t/ha (average) versus 3.11t/ha of unit yield of rice at the ex-post evaluation in 2017, which [almost] maintains the 2012 level. The ex-post showed that “continuous irrigation development in the said site can be considered as the main reason for the increase in land area. Securing an adequate amount of water is an important factor in continuously improving rice productivity.” The research also found that 81% of agricultural incomes as a result of the irrigation had increased, 11% stayed the same, and 8% had decreased. Again, this looks to be among the most resilient projects that, based on ex-post research, included environment which was also found to be as resilient as the livelihoods it was fostering.
  • Sometimes more bad than good news is important when tracking environment and ex-post sustainability: Food for the Hungry, ADRA, and CARE Kenya found that unreliable water supply reduced the motivation to pay for water, threatening the resources to maintain the system [8]. What improved prospects of sustainability understand why communities could not sustain water and sanitation results based on willingness-to-pay models, as well as water being unavailable. Further, a lesson the organizations ideally learned was that “gradual exit, with the opportunity for project participants to operate independently prior to project closure, made it more likely that activities would be continued without project support.” So the question remains, what was learned by these organizations to avoid similar bad results and improve good, resilient results in similar circumstances?

 

[6]

 

Neither sustainability nor environmental quality can be assumed to continue nor to have positive results. Both are extensively under-evaluated, and given climate change disruptions, and this must change. Rowe concludes: “Climate change is a major threat to the long-term sustainability both attacking the natural systems (e.g. lower rainfall or higher floods, worse soil quality, increasing pests attacking crops, disappearing fish stocks, microplastics in our air and water, increasing sea levels from melting glaciers, worsening public health etc.) and destabilizining our Earth’s regenerative capacity. Fortunately, technical barriers do not prevent us from starting to infuse sustainability into evaluation; the barriers are social and associated with the worldview and vision of evaluation.”

 

Sources:

[1] IDEAS 2019 Global Assembly. (n.d.). Retrieved from https://2019.global-assembly.org/

[2] Rowe, A. (2019). Sustainability‐Ready Evaluation: A Call to Action. New Directions for Evaluation, 162, 29-48. Retrieved from https://www.researchgate.net/publication/333616139_Sustainability-Ready_Evaluation_A_Call_to_Action

[3] USAID. (2013, October 31). Environmental Compliance Procedures. Retrieved from https://www.usaid.gov/our_work/environment/compliance/pdf/216

[4] OECD. (2015). Element 4, Paper 1: Global and local environmental sustainability, development and growth. Retrieved from https://www.oecd.org/dac/environment-development/FINAL%20POST-2015%20global%20and%20local%20environmental%20sustainability.pdf

[5] Haraguchi, T. (2017). Socialist Republic of Viet Nam: FY 2017 Ex-Post Evaluation of Japanese ODA Loan Project “Small-Scale Pro Poor Infrastructure Development Project (III)”. Retrieved from https://www2.jica.go.jp/en/evaluation/pdf/2017_VNXVII-5_4.pdf

[6] Coates, J., Kegode, E., Galante, T., & Blau, A. (2016, February). Sustaining Development: Results from a Study of Sustainability and Exit Strategies among Development Food Assistance Projects: Kenya Country Study. USAID. Retrieved from https://www.globalwaters.org/resources/assets/ex-post-evaluation-senegal-pepam

[7] Madagascar Rural Access To New Opportunities For Health And Prosperity (RANO-HP) Ex-Post Evaluation. (2017, June 1). USAID. Retrieved from https://www.globalwaters.org/resources/assets/madagascar-rural-access-new-opportunities-health-and-prosperity-rano-hp-ex-post-0

[8] Kobayashi, N. (2017). Kingdom of Cambodia: FY2017 Ex-Post Evaluation of Technical Cooperation Project: “Technical Service Center for Irrigation System Project – Phase 2 / The Improvement of Agricultural River Basin Management and Development Project (TSC3)”. Retrieved from https://www2.jica.go.jp/en/evaluation/pdf/2017_0900388_4.pdf

 

Assuming Sustainability and Impact is Dangerous to Development (+ OECD/DAC evaluation criteria)

by | Oct 19, 2018 | Doing Development Differently, Ethiopian Red Cross, European Evaluation Society (EES), ex-post evaluation, Exit strategies, foreign aid, Impact, impact evaluation, International aid, international development, NGOs, OECD, OXFAM, Participants, post-project evaluation, SDGs, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Sustained and Emerging Impacts Evaluations (SEIE)

 

Assuming Sustainability and Impact is Dangerous to Development
(+ OECD/ DAC evaluation criteria)

 

We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impact such as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.

INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions” [1]. They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact” [1]. Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer. Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable” [2]. 

How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.

 

 

Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).*

There are remarkable assumptions routinely made. Many projects put sustainability into the proposal, yet most close out projects in the last 6 months. Rarely do projects take the time to properly phase down or phase over (unlike CRS Niger); many exit ceremonially ‘handing over’ projects to country-nationals, disposing project assets, and leaving only a final report behind. Alternatively, this USAID Uganda CDCS Country Transition Plan which looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit [3]. Maybe they will measure progress towards that goal and orient programs toward handover, as in the new USAID “Journey to Self-Reliance” – we hope! Truly, we can plan to exit, but only when data bears out our sustained impact, not when the money or political will runs out.

As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters” [4]. In fact, most ‘impact evaluations’ actually test effectiveness rather than long-term impact. Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.

if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment [5][6]. These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!

 

 

We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants. We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership” [7]. Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms” [7]. It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…

 

 

Footnotes:

[*] This paper is now available at https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/53055

 

Sources:

[1] Simister, N. (2015). Monitoring and Evaluation Series: Outcomes Outputs and Impact. Retrieved from https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Outcomes-Outputs-and-Impact-7.pdf

[2] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved September, 2018, from https://web.archive.org/web/20180919035910/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

[3] USAID. (2016, December 6). USAID Uganda Country Development Cooperation Strategy 2016-2021. Retrieved October, 2018, from https://www.usaid.gov/uganda/cdcs

[4] Porter, S. (2018, October 18). DAC Criteria: The Hand That Rocks the Cradle. Retrieved from https://views-voices.oxfam.org.uk/2018/10/dac-criteria-the-hand-that-rocks-the-cradle/

[5] European Evaluation Society Biennial Conference: Flagship Symposia. (2018). Retrieved from http://www.ees2018.eu/1539782596-flagship-symposia.htm

[6] Ofir, Z. (2018, October 13). Updating the DAC Criteria, Part 11 (FINAL). From Evaluation Criteria to Design Principles. Retrieved from https://zendaofir.com/dac-criteria-part-11/

[7] OECD. (2018, June 11). 2018 Report On The DAC Untying Recommendation. Retrieved from http://www.oecd.org/dac/financing-sustainable-development/development-finance-standards/DCD-DAC(2018)12-REV2.en.pdf

 

What happens after the project ends?  Lessons about Funding, Assumptions and Fears (Part 3)

by | Feb 29, 2016 | 3ie, Accountability, Appreciative Inquiry, Assumptions, CARE, Fears, Food for Peace (FFP), Funding, Learning, microcredit, OECD, OXFAM, PACT, Sustainable development, USAID, Village Banks

 

What happens after the project ends?
Lessons about Funding, Assumptions and Fears (Part 3)

 

In part 1 and part 2 of this blog, we showcased 11 of the 18 organizations that have done post-project evaluations.  While this was scratching the surface of all that is to be learned, we shared a few insights on How we do it Matters, Expect Unexpected Results and Country-national Ownership. We gained some champions in this process of sharing our findings, including Professor Zenda Ofir of South Africa, who said “we cannot claim to have had success in development interventions if the outcomes and/or impacts are not durable, or at least have a chance to sustain or endure.”

 

In this third blog of Lessons Learned from What Happens After the Project Ends, we turn to some of the curious factors that hold us back from undertaking more post project evaluations: Funding, Assumptions, and Fears.

 

Funding

  • Why haven’t we gone back? For the last 2+years Valuing Voices has been researching the issue, we have heard from colleagues: ‘we would love to evaluate post-project but we don’t have any money, ‘donors don’t fund this’, ‘it is too expensive’[*].  Funding currently from bilateral donors such as USAID is given in 1-5 year tranches with fixed terms for completion of results and learning from them and one-year close-out processes [1]. Much of the canon of evaluations conducted after close out that we amassed was from international NGOs that had used their private funds to evaluate large donor-funded projects for their own learning.  Many aimed also to show leadership in sustainability and admittedly dazzle their funders – join them!.
  • We fund capacity building during projects but if we do not return to evaluate how well we have supported our partners and communities to translate this to sustainability, then we fall short. Meetings convened by INTRAC on civil society sustainability are opening new doors for joint learning about factors such as “legitimacy… leadership, purpose, values, and structures” within organizations well beyond any project’s end [2]. The OECD’s DAC criteria for evaluating development assistance define sustainability as: “concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable“ [3]. We need to extend our view beyond typical criterion for sustainability being a focus primarily on continued funding.
  • We need funding to explore whether certain sectors lend themselves to sustainability. In addition to the cases in blog 1, a study by CARE/ OXFAM/ PACT on Cambodian Savings groups finds that we have some revisions to make on how we design and implement with communities to foster sustainability in this sector which typically promises greater sustainability because capital can be recycled [4]. Valuing Voices blogs show indications that once we amass a greater range of post-project evaluations (funders unite), the insights gleaned can illuminate cost efficient paths to more sustained programming, possibly leading to revisions in programming or interventions which have greater likelihood for country-ownership
  • Extend the program cycle to include post-project sustainability evaluation. Rare are donors such as the Australian government (forthcoming) and USAID’s Food For Peace that commission such studies. Rare is the initiative such as 3ie that has research funds allocated by major donors to explore an aspect of impact. We miss out on key opportunities to learn from the past for improved project design if we do not return to learn how sustained our outcomes and impacts have been. We miss learning how we could better implement so more national partners could take on the task of sustaining the changes we catalyzed.
  • We call on donors to fund a research initiative to comprehensively review sustainability evaluations.
  • We call on governments to ask for this in their discussion with donors. 
  • We call on implementers to invest in such learning to improve the quality of implementation today and sustained impact in the years to come.

 

 

Assumptions

Development assistance makes many assumptions about what happens after projects end in terms of people’s self-sufficiency,   partners’ capacity to continue to support activities, and projects’ financial independence and people’s ability to step into the shoes of donors and carry on.   Unless we take a hard look at our assumptions, we will not move from proving what we expect to learning what is actually there.

 

evaluation_-_Hledat_Googlem

 

Among them are these six assumptions:

  • All will be well once we exit; we have implemented so well that of course national participants and partners will be ready and able to carry on without us. We may assume the only important outcomes and impacts are within our Logical Frameworks and Theories of Change. Thus there is no need to return to explore unexpected negative ones, or ways in which the people we strengthened may have innovated in unexpectedly wonderful ways. Aysel Vazirova, a fellow international consultant wrote me: “Post-project evaluations provide data for a deeper analysis of sustainability and help to appreciate numerous avenues taken by the beneficiaries in incorporating development projects into their lives. The theory of change narratives presented by a majority of development programs and projects have a rather disturbing resemblance to the structure of magic tales: (from) Lack – (to) change – (to) happy ending. Post project evaluations have a power to change a rigid structure of this narrative.”
  • We assume evaluations are often used to inform new designs, yet dozens of colleagues have lamented that too often this does not happen in the race to new project design. But there is hope. World Wildlife Fund/UK M&E expert Clare Crawford says when following its new management standards, WWF “expects to see the recommendations of an evaluation before the next phase of design can happen (hence evaluations happen a little before the end of a strategic period).  WWF-UK, when reading new program plans is mandated to verify if – and how – the recommendations of the last evaluation(s) were made use of in the new design phase.  Equally we track management responses to evaluations to see how learning has been applied in current or in future work.” Such a link across the program cycle is not common in our experience and none of the post-project sustained impact evaluations we reviewed said how learning would be used.
  • We may assume data continues accessible from the projects we have evaluated, yet our team member Siobhan Green has found that until recently, with the move toward open data, often project data remains the province of the donors and implementers and to the best of our knowledge leaves the country when projects close. While some sectoral data such as health and education data remains local, we are finding in fieldwork that household level data has been rolled up or discarded once projects close, which makes interviews difficult.
  • We may assume that the participants and partners are not able to evaluate  projects, particularly after the fact.  Being vulnerable does not mean that people are not able to share insights or assess how projects helped or not. Methods such as empowerment evaluation and evaluative thinking are powerful supports [5] [6].
  • Some may assume that the situation has changed in the intervening years, that there is no benefit in returning to see what results remain. Change is inevitable and sometimes more rapid or dramatic than others.  But does that mean we shouldn’t want to understand what happened? This is the greatest disservice of all, for we are selling “sustainable development” so how well have we designed it to be so?
  • We assume that learning for our own benefit is enough. A potential client brought me in to discuss my working on a rare post-project evaluation last year. It was to cost hundreds of thousands of dollars and while would occur in several countries. What I discovered was that while the donor really wanted to learn what results remained more than a decade on, I asked ‘how would the countries themselves benefit from this research and findings?’  There was a long silence. Turns out, nothing from the research would benefit or even remain in country. No one had considered the learning needs of the countries themselves. This simply cannot continue if we are to be accountable to those we serve.

 

Fears

This may be the greatest barrier of all to returning to assess sustainability.

  • We assume our projects continue. We may be afraid to look for what will this tell us about the sustainability of our efforts to save lives and livelihoods so we only choose to publicly study what is successful. Valuing Voices has found that across most the post project studies there is some ‘selection bias’, as we repeatedly learned in our research from colleagues that organizations choose to evaluate projects that are most likely to be successfully sustained.  For instance, USAID Food For Peace’ study notes, “The countries included in this study—Bolivia, Honduras, India, and Kenya—were also chosen because of their attention to sustainability and exit.” Yet as an Appreciative Inquiry practitioner, I would argue that learning what worked best to know what to do more of may be the best way forward.
  • All too often the choice of evaluation design, and sensitivity to findings fly in the face of learning—particularly when findings are negative.  This raises fears around a discontinuation of funding (an implementer fear; a beneficiary fear; could also be a recipient government’s fear). Yet as Bill Gates says, “your most unhappy customers are your greatest source of learning.”>
  • Participants asked during the project cycle about interventions may be fearful of truth telling because of perceived vulnerabilities around promised future resources, local power imbalances in control over resources, or even political imperatives to adopt a particular position. Alternatively we may not believe them, thinking they may not tell us the truth were that to stop resources.

 

Those are ours.

  • Peter Kimeu, my wise advisor and 20-year friend and colleague from Kenya tells us some fears of the that we need to listen to – those that haunt our national partners and participants.

 

They are afraid we do not see their real desires:

  • “It is ‘not how many have you (the NGO) fed, but how many of us have the capability to feed ourselves and our community?’
  • ‘How can we (country national) support our fellow citizens to take our lives and livelihoods into our own hands and excel, sustainably?’
  • ‘What is sustainability if it isn’t expanded opportunities, Isn’t the capability of one to make a choice of value/quality life out of the many choices that the opportunities present?”

 

Will you help us address these challenges? Will you join us in advocating filling the gap in the program cycle, and looking beyond it to how we design and implement with country nationals? Will you, in your own work foster their ownership throughout and beyond? We need to fund learning from sustained impact to transparently discuss assumptions and face our fears.  This is a sustained purpose we need to and can fill.

 

 

Sources:

[1] Capable Partners Program & FHI 36. (2010). Essential NGO Guide to Managing Your USAID Award: Chapter 6 – Close Out. Retrieved from https://www.ngoconnect.net/sites/default/files/resources/Essential%20NGO%20Guide%20-%20Chapter%206%20-%20Close%20Out.pdf

[2] Hayman, R. (2014, November 5). Civil society sustainability: Stepping up to the challenge. Retrieved from https://www.intrac.org/civil-society-sustainability-stepping-challenge/

[3] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved 2015, from https://web.archive.org/web/20151206171605/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

[4] Emerging Markets Consulting. (2013). Sustainability of Savings Group Programs in Cambodia for CARE, Oxfam, and Pact. Retrieved from https://mangotree.org/Resource/Sustainability-of-Savings-Group-Programs-in-Cambodia-for-CARE-Oxfam-and-Pact

[5] Better Evaluation. (n.d.). Empowerment Evaluation. Retrieved from https://www.betterevaluation.org/plan/approach/empowerment_evaluation

[6] Griñó, L., Levine, C., Porter, S., & Roberts, G. (Eds.). (2016). Embracing Evaluative Thinking for Better Outcomes: Four NGO Case Studies. Retrieved from https://www.theclearinitiative.org/resources/embracing-evaluative-thinking-for-better-outcomes-four-ngo-case-studies

 

[*] It does not have to be. We have done these evaluations for under $170,000, all-inclusive.