Blog

Upcoming Jan Webinar: Lessons from Nordic / the Netherlands’ ex-post project evaluations: 14 Jan 2021

Posted by on Jan 9, 2021 in Denmark, Evaluation, ex-post evaluation, Finland, impact evaluation, Netherlands, Nordics, Norway, Sustainability, Sustainable development, Sweden, Valuing Voices | 0 comments

Upcoming Webinar: Lessons from Nordic / the

Netherlands’ ex-post project evaluations: 14 Jan 2021

In June-August 2020, Preston Stewart, our Valuing Voices intern, conducted through government databases of the four Nordic countries – Norway, Finland, Sweden, and Denmark – and the Netherlands to identify ex-post evaluations of government-sponsored projects. The findings and recommendations for action are detailed in a four-part white paper series, beginning with a paper called The Search.

We would like to invite you to a presentation of our findings, and a discussion of what can be learned about:

1) the search process,

2) how ex-post evaluations are defined and categorized,

3) what was done well by each country’s ex-posts,

4) sustainability-related findings and lessons, and

5) what M&E experts in each country can improve on ex-post evaluation practices.

One big finding is that there were only 32 evaluations that seemed to be ex-post project, and only 1/2 of them actually were at least 2 years after project closure.

 

We have many more lessons about conflicting definitions, that ex-post evaluation is not the norm in the evaluations processes of the five governments, that development programs could, if committed to ex-post evaluation, learn about sustainability by engaging with the findings from many more such evaluations, and to increase accountability to the public and for transparent learning, ex-post evaluations should be shared in public, easy-to-access online repositories.

Join us!

REGISTER: https://www.eventbrite.com/e/webinar-lessons-from-nordic-the-netherlands-ex-post-project-evaluations-tickets-132856426147 

Your ticket purchase entitles you to the webinar, its meeting recording, associated documents, and online Sustainability Network membership for resources and discussion. Payment on a sliding, pay-as-you-can scale.

Interview repost: Why Measure & Evaluate Corporate Sustainability Projects?

Posted by on Dec 18, 2020 in CSR, Evaluation, Sustainability, Sustainable development | 0 comments

Interview repost: Why Measure & Evaluate Corporate Sustainability Projects?

A CSR firm in Central Europe asked me to talk about sustainability, evaluation, and Corporate Social Responsibility. We had a terrific interview – please click thru to: https://www.besmarthead.com/cs/media/post/smartheadtalks-2-jindra-cekan-eng

 

 

The topics we discussed include:

* Sustainability in CSR involves value creation through benefits-creation for both companies and consumers

* Projects feeding into the United Nations Sustainable Development Goals may not be sustained over the long-term and we must return to evaluate what could be sustained and what emerged ex-post

* Scale measurable and meaningful impact investment through the transformation of development nonprofits’ programming + approach

* Three activities in sustainability that companies can start doing right away, that you would recommend to any company anywhere in the world

And Very Happy Holidays to all… may our lives and world be more sustainable in 2021….

 

 

Reblog: ITAD/CRS “Lessons from an ex-post evaluation – and why we should do more of them”

Posted by on Dec 10, 2020 in Catholic Relief Services (CRS), Credit, Evaluation, ex-post evaluation, microcredit, Microenterprise, Sustainable development | 0 comments

Reblog: ITAD/CRS “Lessons from an ex-post evaluation – and why we should do more of them”

Reposted from: https://www.itad.com/article/lessons-from-an-ex-post-evaluation-and-why-we-should-do-more-of-them/

Even as evaluation specialists, rarely do we get the chance to carry out ex-post evaluations. We recently carried out an ex-post evaluation of Catholic Relief Services’ (CRS) Expanding Financial Inclusion (EFI) programme and believe we’ve found some key lessons that make the case for more ex-post evaluations.

We’ll be sharing learning from the evaluation alongside CRS colleagues at the Savings Led Working Group session on Members Day of the SEEP Annual Conference – so pop by if you would like to learn more.

What is an ex-post evaluation?

Ex-post evaluations are (by definition) done after the project has closed. There is no hard and fast rule on exactly when an ex-post evaluation should be done but as the aim of an ex-post evaluations is to assess the sustainability of results and impacts, usually some time will need to have passed to make this assessment.

A little bit about EFI

EFI was a Mastercard Foundation-funded program in Burkina Faso, Senegal, Zambia and Uganda whose core goal was to ensure that vulnerable households experienced greater financial inclusion. Within EFI, Private Service Providers (PSPs) formed and facilitated savings groups using CRS’ Savings and Internal Lending Communities (SILC) methodology, with the SILC groups responsible for paying the PSP a small fee for the services that they provide.

This payment is intended to improve sustainability by incentivising the groups’ facilitators to form and train new groups, as well as providing continued support to existing groups, beyond the end of the project.

A little bit about the evaluation

So, if the aim of the PSP model is sustainability, you need an evaluation that can test this! Evaluation at the end of project implementation can assess indications of results that might be sustained into the future. However, if you wait until some time has passed after activities have ended, then there is much clearer evidence on which activities and results are ongoing – and how likely these are to continue. Uganda was also a great test case for the evaluation because CRS hadn’t provided any follow-on support.

Our evaluation set out to assess the extent to which the EFI-trained PSPs and their SILC groups were still functioning 19 months after the programme ended and the extent to which the PSP model had contributed to the sustainability of activities and results.

What the ex-post evaluation found

We found a handful of findings that were only possible because it was an ex-post evaluation:

  • There were 56% more reported groups among the sampled PSPs at the time of data collection than there were at the end of the project.
  • Half of the PSP networks established within the sample are still functioning (to some extent).
  • PSPs continued to receive remuneration for the work that they did, 19 months after project closure. However, there were inconsistencies in frequency and scale of remuneration, as well as variation in strategies to sensitize communities on the need to pay.

This only covers a fraction of the findings but we were able to conclude that the PSP model appeared to be highly sustainable. The evaluation also found that there were challenges to sustainability which could be addressed in future delivery of the PSP model. Significantly, the PSP model was designed with sustainability in mind – and this evaluation provides good evidence that PSPs were still operating 19 months after the end of the project.

What made the evaluation possible

We get it. It isn’t always easy to do ex-post evaluations. Evaluations are usually included in donor-implementer contracts, which end shortly after the project ends, leaving implementers without the resources to go back and evaluate 18 months later. This often results in a lack of funding and an absence of project staff. This is also combined with new projects starting up, obscuring opportunities for project-specific findings and learning as it’s not possible to attribute results to a specific project.

In many ways, we were lucky. Itad implements the Mastercard Foundations Savings Learning Lab, a six-year initiative that supports learning among the Foundation’s savings sector portfolio programmes – including EFI. EFI closed in the Learning Lab’s second year and with support from the Foundation and enthusiasm from CRS, we set aside some resource to continue this learning post-project. So, we had funding!

We also worked with incredibly motivated ex-EFI, CRS staff who made time to actively engage in the evaluation process and facilitate links to the PSP network, PSPs and SILC group members. So, we had the people!

And, no-one had implemented a similar PSP model in supported districts of Uganda since the end of EFI. So, we were also able to attribute!

Why we should strive to do more ex-post evaluations

Despite these challenges, and recognising it isn’t always easy, doesn’t mean it is not possible. And with projects like EFI where sustainability was central to its model, we would say it’s essential to assess whether the programme worked and how the model can be improved.

Unfortunately, practitioners and evaluators can shout all we like but the onus is on funders. We need funders to carve out dedicated resource for ex-post evaluations. This is even more important for programmes that have the development of replicable and sustainable models at their core. For some projects, this can be anticipated – and planned for – at project design stage. Other projects may show promise for learning on sustainability, unexpectedly, during implementation. Dedicated funding pots or call-down contracts for ex-post evaluations are just a couple of ways donors might be able to resource ex-post evaluations when there is a clear need for additional learning on the sustainability project results.

This learning should lead to better decision making, more effective use of donor funds and ultimately, more sustainable outcomes for beneficiaries.”

Other Findings:

Some of the other findings of this report on Financial Inclusion are:

RESOURCES: “Finding 1.iii. PSPs continue to receive remuneration for the work that they do; however, there are inconsistencies in frequency and scale of remuneration, as well as variety in strategies to sensitize communities on the need to pay.”

CAPACITIES: “Finding 3.ii. All networks included a core function of “collaboration, information-sharing and problemsolving”; however, networks were not sufficiently supported or incentivized to fulfill complex functions, such as PSP quality assurance or consumer protection, and their coverage area and late implementation limited the continued functioning of networks.”

PARTNERSHIP: “Finding 2.i. Only four of the 24 groups are clearly linked with other stakeholders and two were supported by EFI to create these linkages.”

Consider doing one!

What makes it Difficult to Evaluate Projects Ex-post: Lessons from WWF 

Posted by on Nov 20, 2020 in environment, ex-post evaluation, Sustainable development | 0 comments

What makes it Difficult to Evaluate Projects Ex-post: Lessons from WWF 

 

The World Wildlife Fund (WWF) generously shared lessons from two ex-posts and in particular what they felt should not have been done in terms of organizational learning. They originally wanted to see whether ”the work done is sustainable and so to provide lessons for the implementing office, donor offices and the wider WWF network about designing and implementing projects to deliver outcomes that are sustained beyond WWF’s involvement.

They used six criteria for country/ case selection:

 

This was in line with Valuing Voices’ (Ex-post) Evaluability Checklists.

  • Organizational considerations including interest in learning by WWF
  • Methodological considerations regarding data access, quality etc.
  • Choice of projects, sites, timing to ensure post-closure, contribution of results isolated

 

 

WWF Case 1:

The first WWF case examined the importance of evaluating both Sustained and Emerging Impacts (those expected or those that emerged from local efforts or other entrants post-INGO-exit) and managing the ex-post evaluation process, as LWR and World Vision have done.

There were three types of interesting results (refer to Table 1).

  • Methodological problems in comparability to final evaluation and impact data as well as choice of project: problems with site/ case and lack of water data comparability as well as the entry of a new NGO garnering benefits (so limited contribution/ isolation of results). WWF tried to mitigate it with good design guidance and ongoing review processes during the ex-post.
  • The emergence of new data ex-post about the benefits of the local user association for conflict management and emerging new catchment management
  • There were ex-post limitations intrinsic to the species and ecosystem resources on which they rely, specifically species and water. Other problems of not tracking impacts on animals, not learning from evaluations, and not having saved documents might benefit other projects’ quality.

Table 1. WWF Case 1

Some learning might be attributable to the difficulties of evaluating ecosystems or it might provide important cautionary lessons for others considering such ecosystem evaluations.

But overall, it shows that in contexts where one of these criteria are present:

1. original programme having design flaws,

2. poor collection of evidence of progress against the chosen indicators of success,

3. poor documentation retention,

Then, there is limited value to be gained from doing an ex-post evaluation.

 

WWF Case 2:

The excellent news is that ex-post demonstrated that all three kinds of impact indicators, namely species numbers, forest cover, and conservation area improved (note, data are confidential, thereby not cited below) and that the XX Conservation Area is still managed by the community (Table 2). Also, the team gathered information that living standards had improved, as had, impressively a country-level seed fund and active participation in managing and scientifically learning from their resource.

The second case also highlights typical ex-post methodological evaluation considerations, including choice of sites/ timing:

  1. not applying a standard monitoring process for the whole programme period, which caused confusion about end of project versus ex-post impact progress
  2. the ex-post evaluation was planned predicated on the project finishing and staff exit from the area, but there was ongoing financial support of the CSO from the central office for a further two years which meant that this was not a true ex-post evaluation as it happened before the two-year minimum timeframe
  3. measurement led to ‘anecdotal’ evidence and comparisons made to national statistics rather than change from project evaluations which limits ex-post learning particularly
  4. there were negative findings of sustainability how well the community appreciated the XXCA and insufficient economic return on investment expectations that some costs such as trash removal were not covered by anticipated tourist revenues.

It also highlights that ecosystem ex-post evaluation is complex, as wildlife number was difficult to methodologically confirm, due to progression in monitoring technologies.  A valuable question that WWF shared was “how to assess [continued] benefits to people from natural resource management?”

 

Table 2: WWF Case 2

WWF found and shared important and appropriate lessons about what these ex-posts brought with them. Most fell into aforementioned categories of Choice of projects, sites, timing, and importantly, the value of considering Methodological considerations regarding data access, quality from project onset.

What was new to us at Valuing Voices was that Organizational considerations including interest in learning by WWF-UK and their partners which is something we have not seen before.  Equally, the importance of a solid rationale for asking a country office to participate in such an evaluation, and not imposing an ex-post on a less than enthusiastic program for whom sustainability may not be a priority.  This links these cases together, as given limited budgets, WWF and their partners could choose to prioritize design spending, perhaps above final evaluations, especially if local NGOs know donors will leave at project’s end, so funding to improve their internal capacity strengthening might be a larger priority. Knowing the project has an immutable closeout can affect interest at the headquarters, as it can be quite difficult to find funding to evaluate ex-post, and more pressing current work competes. Also, implementing local NGOs/ CBOs could actually know about sustainability far more than donors, hereby eliminating their interest, as an M&E officer recounted. Both organizations could have other priorities, including new projects with new staff, who had less interest in the old project.  Staff turnover is an additional, important consideration in whether to do an ex-post.

 

Table 3: WWF Learning

 

Given how vital it is to evaluate the sustainability of our environment, including ecosystems and natural capital on which our participants and partners depend, WWF is going ahead on evaluating ecosystems, adding to the vital knowledge base. The Global Environmental Facility is also experimenting on remote-monitoring for the evaluation of ecosystems. USAID’s Water Bureau has also done ex-posts on Water/ Sanitation/ Hygiene.

We have a whole world of sustainability to discover!

Interactive Webinar: Sustained Exit? Prove it or Improve it! (Nov 6 2020)

Posted by on Oct 28, 2020 in Aid effectiveness, Evaluation, Impact, measuring sustainability checklist, post project evaluation, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluations (SEIE) | 1 comment

Sustained Exit? Prove it or Improve it!

(reposted from Medium https://jindracekan.medium.com/sustained-exit-prove-it-or-improve-it-702ac507e2a5)

Do we exit global development projects knowing our impacts are sustained? We hope so. As Professor Bea Rogers of Tufts said after evaluating 12 projects 2 years post-closure ( https://www.fsnnetwork.org/resource/exit-strategies-study), “ Hope is Not a Strategy”, yet too often that is what projects that assume sustainability does. They/we hope. But is this good enough? For me, confirming that hope means evaluating beyond exit to ex-post, at least 2 years after donor investments end. 99% of the time, donors & development practitioners don’t return to see what lasted, what didn’t, why nor what emerged from people’s own effort. Yet we implement similar programs over and over onward, not learning lessons from the past. Sigh.

We need to evaluate what we expected to remain from our implemented projects. We also need to learn from what evaluator Bob Williams calls, “the sustainability of the idea that underpinned the results (even if the results were no longer evident)”. This is often beneath what emerged: Our projects catalyzed the local’s desire to sustain activities: taking new ways, that are locally manageable (changing how the development idea is implemented onward) or even having entirely new initiatives emerge from the participant groupings — from their own priorities, not ours. (For more on emerging impactshttps://www.betterevaluation.org/en/themes/SEIE)

Evaluation leaders talk about power, they talk about the environment. After 7 years of researching and evaluating projects ex-post evaluations, I have found there are no brilliant 100% sustained + projects nor are there any 100% abjectly scorched earth ones either. Our results are middling at best. And therein lies the rub. Projects are what donors want to give. Sometimes that overlaps with what recipient countries want, sometimes not. Most of the time the resources to sustain our multimillion-dollar, -euro, -yen, etc., investments aren’t there. We can use incentives (e.g. food aid or cash) that can bolster short-term success while we spend, but once phased out, can lead to sustainability sharply falling off as early as 2 years after we exit. It’s because while ‘development’ is about ‘our’ spending on ‘our’ programs, about short-term success while we’re implementing, rather than our equal partners’ priorities and ability to sustain it. We misuse our power. We care about ourselves far more than the people we ostensibly went there to ‘save’.

And as esteemed evaluators Andy Rowe/ Michael Quinn Patton noted, given climate change we need to question even more assumptions about how sustained and resilient our programming can be, by evaluating the natural environment on which our programming relies pre, during & implementation, at exit and ex-post closure. (More on sustained environment: https://valuingvoices.com/sustaining-sustainable-development/)

It also means we need to talk to those to whom we will eventually hand over early on to make sure we’ve built-in resilience to the climatic, economic shocks we know of so far. I recommend my colleagues Holta Trandafili and Isabella Jean’s presentations on partnering we did a couple of months ago: https://valuingvoices.com/sustainability-ready-what-it-takes-to-support-measure-lasting-change-webinar/

Finally, I have come to see that to make sustainability more likely for years to come, we must fund, design, implement, and monitor/ evaluate For Sustainability throughout the project cycle. I have come to see that folks need guidance to help support their integration of sustainability throughout, including environment & resilience, benchmarks, and more. We can learn from what ex-posts teach. Join me please, to help craft more sustained development:

Upcoming Sustained Exit Webinar: 6 Nov 2020, 14:30–17 CEST, 8:30–11 EST

“Sustained Exit? Prove it or Improve it!” Interactive webinar discussion of ex-post sustainability evaluation lessons and how to integrate into ongoing #aid programs. On Zoom, participants get resources: checklists, slides, recording, Join us to #sustain #impacts! Register, sliding scale: https://sustainedexit.eventbrite.com

Sustainability Ready: what it takes to support & measure lasting change webinar

Posted by on Jul 7, 2020 in Accountability, ex-post evaluation, International aid, Participation, post-project evaluation, Sustainable development | 2 comments

On June 24th under GLocal’s UNConference, “Co-creating our future stories of hope and action”, Jindra Cekan, Holta Trandafili, and Isabella Jean presented their work on sustainability evaluations and exit strategies via local voices. We chaired a 2-hour discussion session on the following topics:

  • Sustainability of global development projects and exit from them,
  • The importance of valuing local partners’ and participants’ voices,
  • How to embed ex-post evaluation of sustainability into the project cycle,
  • How expectations, benchmarking, and early joint planning in exit strategies, as well as considering long-term ownership & relevance will support projects to be sustained locally, including questioning who will maintain results,
  • Considering power dynamics between donors and ultimate ‘beneficiaries’ and
    the value of the impact of the project from a variety of perspectives, and more

Here is the recording of our presentation or see just the PowerPoint presentation. We harvested lessons from our three presentations:

Jindra Cekan:
· Fear of learning about failure in our global development industry – INGOs are “waiting for a successful enough project” to commission ex-post project sustainability evaluation
· It needs to be a culture of learning, not a culture of success.
· Lack of transparency in sharing program evaluation results with communities and local government is widespread, and even more rarely do we come back after many years and share learnings
· Participatory approaches are vital, listening to participants about sustained impacts is key
· It is never attribution, always contribution. To isolate impacts, we need to look for project sites that haven’t had multiple other organizations overlapping through all phases.
· Building sustainability planning throughout the project cycle is key – but often doesn’t happen

Holta Trandafili:
· Sustainability needs to be planned to be researched, including evaluating why or why not were project elements sustained, and why? What has the project done to enable communities to sustain improvements?
· Expectations of sustainability need to be more modest (as most results are mixed good/bad)
· We need to ask: How are you defining and measuring sustainability – for how long should the results last? Among how many participants? Have you set benchmarks for success?
· We should expand your toolbox on methodology to investigate sustainability. Stories of success are one of a myriad of methods used, including mixed-methods, cost-benefit, etc.
· Start with the need for learning not [just] accountability

Isabella Jean:
· Sustainability investigation/evaluation/learning should be mindful that this is NOT about projects. It is about people.
· We have a system that focuses on gaps and needs to be filled vs. existing capacities’ structures to be reinforced. How can our work on measuring sustainability bring this to light and call it out, so that we change the norm?
· Planning for sustainability requires the insight to integrate resources and experiences of outsiders with the assets and capacities of insiders to develop context-appropriate strategies for change

If you would like to discuss this with any of us, please send us comments and we’re happy to respond. Thanks again to the G-Local UnConference team!

 

 

 

 

 

 

Below please find our bios:

Jindra Cekan/ova has worked in global development for 33 years focused on participatory design and M&E for global non-profits. She founded Valuing Voices 7 years ago. For details, see: Valuing Voices Founder

Holta Trandafili is the Research, Learning, and Analytics Manager with World Vision US and has been leading field research, monitoring, and evaluation since 2007. She has led sustainability measurement studies for World Vision programs in Uganda, Kenya, Sri Lanka, India, Burma, and Bolivia. Her areas of expertise and interest include program and community groups’ sustainability measurements; agency-level measurements; empowerment approaches to development; integrated programming; local capacities for peace; gender analysis; and outcome monitoring. Currently, Mrs. Trandafili serves as an Advisory Committee member for InterAction’s Effectiveness and Program Evaluation Working Group and chairs one of the sub-working groups under The Movement for Community-led Development.

Isabella Jean supports international and local organizations and funders to document promising practices, facilitate learning and strengthen capacities for conflict sensitivity, peacebuilding and humanitarian effectiveness. She has facilitated action research, collaborative learning and advisory engagements in over 25 countries, and serves as an advisor to policymakers, senior leadership and program teams. Isabella co-authored the bookTime to Listen: Hearing People on the Receiving End of Aid and developed practical guidance to support accountability to communities, listening and feedback loops, and responsible INGO exits. She teaches graduate-level courses on aid effectiveness, program strategies and M&E of peacebuilding at Brandeis University’s Heller School for Social Policy and Management. Previously, Isabella directed training at a community organizing network and conducted policy research for the Institute for Responsive Education, UNDP, and Coexistence International.

 

Sustainable Exit Strategies: USAID vs. EU

Posted by on Jun 28, 2020 in EU, Exit strategies, USAID | 0 comments

Sustainable Exit Strategies: USAID vs. EU

 

Sustainable Exit Strategies: USAID vs. EU

 

Once malnutrition has decreased, students’ attendance has risen, or the number of small businesses has doubled, program implementers may be quick to pack their bags and leave. But the impact of their work may be undermined if not undone if they leave before implementing context-specific, comprehensive plans for post-project sustainability. It is for this reason that exit strategies – and particularly institution-wide mandates by donors – are crucial components of effective development programming. The United States Agency for International Development (USAID and the European Union (EU) Commission both have close-out protocols in place, but much to learn from one another in ensuring long-term progress in the wake of their close-outs.

 

Recommendations for Project Close-Out

Although every program should develop and follow context-specific exit measures, non-profits and ex-post evaluators have found that some exit strategies consistently lead to long-term, sustainable impact. The International NGO Training and Research Centre (INTRAC), for example, states that donors and implementers should actively plan for close-out throughout the project – from program design to the post-exit stage [1]. It recommends the following:

  • Implementers should “build exit thinking into the design of the project,” and during the program, representatives should be monitoring pre-determined exit indicators [1]
  • “Senior staff and management need to prioritize resources for exits in order to do them well, embed learning within organizations and across teams, and ensure mistakes are not repeated” [1]
  • “Practitioners should also consider going the extra mile to look after staff at headquarters and within partner organizations” [1]
  • “Document experiences and share them externally” [1]
  • “Consider whether a timeline approach would work for you” [1]

 

The USAID Approach to Project Close-Out

According to USAID’s 2020 Program Cycle Operational Policy, the formal close-out process for specific projects are carried out by the USAID Mission team in the recipient country and Operating Units based in Washington, DC [2]. However, the Program Cycle does not reference the close-out process, let alone offer guidance [3]. Unlike country exits, which require thorough evaluation and international cooperation [4], the sustainability measures of close-out procedures for activities – components of country-wide programs – seem to be delegated to implementing agencies.

Tellingly, the USAID formally defines “sustainability” as “the ability of a local system to produce desired outcomes” after its projects end, which the Agency may “contribute to” by strengthening capacity in their respective realms [5]. After close-out, USAID currently considers the sustainability of its projects to be a matter of “the country and/or targeted community’s commitment and capacity to achieve development solutions” [5]. Commitment and capacity, in turn, are gauged by 17 country-level “self-reliance metrics” ranging from “liberal democracy” to “export sophistication” [6,7]. Even post-project monitoring and evaluation are explicitly meant to “reveal whether implementation is on track and results are being achieved” with no mention of sustainability or long-term follow-up [3].

In effect, this framework not only puts the onus of project sustainability on recipients, but evaluates it based on large-scale and potentially tangential metrics. The stakes of this practice are raised by the fact that the resultant “Country Roadmaps” inform when USAID ends its programming in a given country, endangering a range of populations that these metrics do not account for, like seniors and prisoners [6].

 

USAID’s official Program Cycle [3]

 

During close-out, USAID representatives seem to focus on administrative tasks. When a Mission is preparing for activity close-out, for example, Mission Directors are responsible for developing plans that include the following:

  • “The retention of sufficient and suitable staff members and the delegation of authority and assignment of specific responsibility to each to carry to completion the required close-out action” [8]
  • “Estimates of the personnel, space, and funds required to complete the close-out” [8]
  • “The timetable for phasing out, transferring, or terminating U.S. direct-hire, participating agency, and contract personnel and the replacement of assistance from other resources” [8]
  • “Actions for terminating the services of cooperating country and third country national staff” [8]
  • In its 2003 version, but since removed:
  • “Missions must think in terms of leaving sustainable and useful units of assistance” [9]
  • “A monthly timeframe/activity projection for the entire close-out period [approximately six months] of major activities that must occur” [9]

Similarly, in the case of Interagency Agreements, USAID Agreement Officers’ (AO) personal and delegated close-out responsibilities are limited to the collection of financial reports and compliance-related documents [10]. AOs are expected to work in this same, limited capacity as they terminate Assistance and Acquisition Awards [11].

By contrast, when USAID gives grants to NGOs and has “little direct involvement,” recipients are strongly encouraged to adhere to six substantial, well-rounded “tips,” which include [12,13]:

 

A List of Recommendations, including "Work with Staff to ensure a smooth transition"

*Staff refers to each NGO’s own staff; based on [13]

 

Among all of the terms and conditions reviewed, USAID’s exit strategies for foreign Missions, Agreements, and Awards have been almost exclusively administrative in nature. Although the Agency should not be solely responsible for ensuring the sustainability of its programming, its lack of regulation in this realm risks both undue pressure on implementing partners to do so and short-lived progress in the wake of its exit [14].

Fortunately, USAID has demonstrated its interest in sustainable close-out by commissioning studies and reports. In March 2020, USAID’s three-year “collaborative learning project” called Stopping as Success published a checklist for sustainable project transition [15,16]. The publication offers 16 concrete suggestions ranging from “build[ing] trust” with local communities to requiring implementers to report their progress in carrying out a pre-determined sustainability plan [16]. Similarly, the Food and Nutrition Technical Assistance III Project – a partnership between USAID and FHI 360 – studied the consequences of four food assistance programs and made thorough recommendations for ensuring the long-term impact of future projects [14]. These recommendations include accounting for “predictable external shocks and stressors” like natural disasters, and building “vertical linkages […] between community-based organizations or individuals and existing public or private sector institutions” [14]. Armed with this impressive base, USAID is well-equipped to account for new, evidence-based close-out practices.

UPDATE: A NEW DISCUSSION NOTE BY USAID ON EX-POST EVALUATION WHERE EXIT IS MENTIONED.

The EU Approach to Project Close-Out

Given the sheer scale of EU development programming – 28 countries providing 47% of the world’s development aid – no specific, all-encompassing mandates for the close-out process was found [17]. Perhaps as a result, there is notoriously little development policy coordination among Member States:

  • Due to inconsistencies among the eight documents outlining the EU’s development objectives and indicators, an internal audit characterized its collective approach as “mixed – at best” [18]
  • High-level meetings have explicitly prioritized “consensus on development” and “working better together,” producing appropriately vague guidelines that do not specifically address closure [19,20]
  • e.g. EU staff should “help to promote more inclusive, sustainable growth that does not compromise the ability of future generations to meet their needs” [19]
  • Its main development finance body, the Development Cooperation Instrument, has been critiqued for its “ambitio[us] desig[n] to address a very broad range of issues, from policy priorities in the agenda for change to EU policy concerns” [21]

For insight into the EU’s exit strategies, one must look to guidelines for specific types of development programming.

 

Cover of the New European Consensus on Development

[33]

 

In the case of joint programming, when Member States work together to design and implement a country-wide, multi-sectorial project, donors must “reach agreement on a responsible exit strategy” for “phasing out sectors outside focal/centration areas, without creating a financing gap and with minimum disruption to the partner country” [20]. The policy offers two examples of such a strategy: “including scenarios in which other participating partners ‘plug the gap’ and/or other forms of cooperation beyond ODA [official development assistance] (e.g. the private sector)” [20]. The strategy must be drafted with input from the recipient country “as much as possible,” and may include either the intervention of other stakeholders or the private sector [20]. These discussions make take place at any of the following four “entry points” [20]:

  • “At the start of a new national development plan cycle, or a major change in the country” [20]
  • “At the start of a new strategic programming period or a review process for several important development partners” [20]
  • “During high-level country meetings […] where partner governments present their plans and solicit development partners support” [20]
  • “In situations of fragility or transition from humanitarian aid to more structured programmatic aid and more development partners” [20]

The development of context-specific exit strategies by Member States and recipients allows for practicable, fairly delegated, and concrete steps to exiting. However, this provision omits mention of enforcement or evaluation, may take place at the end of the implementation period, and does not encompass best practices as defined by INTRAC [1].

Outside of joint programming, the EU oversees projects funded by its executive implementing agencies, such as the Consumers, Health, Agriculture and Food Executive Agency and the Executive Agency for Small and Medium-sized Enterprises (EASME). In those cases, agency-wide final report guidelines could shed light on collective guidance but these are publicly inaccessible [22]. However, an early iteration of the Horizon 2020 Programme’s final report guide is still available online. The EU had required that implementers submit 20-30 lines of text addressing the following:

“Inform about activities undertaken during the project to ensure that your action
continues and reaches out further. Inform also about plans and intentions to continue
and sustain the project activities after the end of the project. Indicate the state
of advancement (e.g. agreed action, concrete proposal, idea)” [23].

Although field staff may not have been required to tangibly ensure the sustainability of their projects – and may not even make these assessments anymore – it is clear that at one time, the EU mandated assessments of expected sustainability at the end of program-specific project implementation.

With no overarching guidance for EU project exits, countries and individual NGOs have resorted to choosing or developing their own. The think tank E3G – funded in part by EASME [24] – has issued both internal and guidance for sustainable programming [25,26]. Its key points include “consultative engagement[s with] a broad range of stakeholders,” “integrat[ing programs] with long term national development plans,” and “strengthening city and local-level planning and institutions” [25]. These recommendations, however, are scattered among legal studies, commissioned reports, and briefing papers. By contrast, another EASME-funded program called Carbon Market Watch adheres to the project cycle guidelines of UN committees [27,28,29]. Although their Project Design Document and methodology forms request specific information about monitoring and evaluation, they do not ask for implementers’ plans for program close-out or long-term impact [30].

 

Cover of People in Need Evaluative Report

[32]

 

Other EU member-country NGOs end up developing sustainable exit measures mid-project. One example is a 2011-2013 agricultural project in Ethiopia implemented by People in Need, an international NGO based in the Czech Republic and funded in part by the EU’s Commission for Civil Protection and Humanitarian Aid [31]. Two months before its end, an evaluation of the program’s sustainability was conducted, receiving such recommendations as: 1) “increasing [recipients’] financial sustainability;” 2) “ensuring that FTCs’ [Farmer Training Centers] services benefit all, not only model farmers;” and 3) “providing books in Amharic language, not English” [32]. In the remaining time, 75% of these recommendations were met, leading to the internal assessment that the “sustainability of the project’s outcomes is likely to be largely positive” [32].

 

Learning from One Another

USAID and the EU operate at starkly different scales – one a country-specific bilateral donor, and the other an economic union of dozens of donor countries – but each of their exit strategies could be strengthened by incorporating elements from the other’s.

The EU could learn from the USAID’s multifaceted guidance for NGOs; the considerations listed in its final report guide are limited to the post-project involvement of EU field staff. It should explicitly promote local participation for sustainability during project close-out and beyond, such as transferring operations to local organizations or businesses.

In turn, USAID’s administration-centric strategy could benefit from the large-scale considerations of EU policies. Discussions with recipient governments, particularly in turbulent circumstances, could strengthen diplomatic relations with the US in addition to supporting reasonable transition plans. Formally soliciting exit plans from project implementers could also produce more sustainable project results.

 

NOTE: Valuing Voices thanks our excellent Harvard intern Rachel Sadoff for this research! We have a question: we found no EU/EC sites discussing exit strategies recommendations or guidance for global development projects. Do you have some? Please share in the comments below. Thank you.

Footnotes:

A) Projects and activities need to explain and demonstrate how their programming advances the objectives of USAID’s Country Development Cooperation Strategies for the given state (see ADS Chapter 201).

B) Current totals are publicly available on the Agency’s Country Roadmap website.

C) VV’s search spanned over five hours.

D) As of writing, final report guidelines are only available through an exclusive portal.

E) These are the Clean Development Mechanism and Joint Implementation Committee of the United Nations Framework Convention on Climate Change (UNFCCC).

 

Sources:

[1] Lewis, S. (2016, January). Developing a Timeline for Exit Strategies. Retrieved from https://www.intrac.org/resources/praxis-paper-31-developing-timeline-exit-strategies/

[2] USAID. (2020, June 16). ADS Chapter 201: Program Cycle Operational Policy. Retrieved from https://www.usaid.gov/sites/default/files/documents/1870/201.pdf

[3] USAID. (2016, December 8). ADS 201: Program Cycle Operational Policy – Fact Sheet. Retrieved from https://usaidlearninglab.org/library/ads-201-program-cycle-operational-policy-fact-sheet

[4] USAID. (2020, February 26). Strategic Transitions Fact Sheet. Retrieved from https://www.usaid.gov/documents/1870/strategic-transitions-fact-sheet

[5] USAID. (2020, May). Discussion Note: Ex-Post Evaluations. Retrieved from https://usaidlearninglab.org/library/discussion-note-ex-post-evaluations

[6] USAID. (2020, June 3). USAID’S Country Roadmaps Fact Sheet. Retrieved from https://www.usaid.gov/documents/1870/usaids-country-roadmaps-fact-sheet

[7] USAID. (n.d.). The Journey To Self-Reliance: Country Roadmaps. Retrieved June 2020 from https://selfreliance.usaid.gov/

[8] USAID. (2018, July 17). Mandatory Reference for ADS Chapter 527: Administrative Guidance on How to Close a USAID Operating Unit – Checklists. Retrieved from https://www.usaid.gov/sites/default/files/documents/1868/527mab.pdf

[9] USAID. (2003, October 1). ADS 527 Mandatory Reference: Guidance on How to Close a USAID Mission – Checklist. Retrieved from https://2012-2017.usaid.gov/sites/default/files/documents/1868/527mab.pdf

[10] USAID. (2015, November 30). ADS Chapter 306: Interagency Agreements. Retrieved from https://2012-2017.usaid.gov/sites/default/files/documents/1868/306.pdf

[11] USAID. (2010, October 6). Guidance on Closeout Procedures for A&A Awards. Retrieved from https://2012-2017.usaid.gov/sites/default/files/documents/1868/302sat.pdf

[12] USAID. (2020, February 13). Doing Business with USAID. Retrieved from https://www.usaid.gov/work-usaid/how-to-work-with-usaid

[13] USAID, & FHI 360. (2012). The Essential NGO Guide to Managing Your USAID Award. Retrieved from https://www.fhi360.org/resource/essential-ngo-guide-managing-your-usaid-award

[14] Food and Nutrition Technical Assistance. (2015, December). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp

[15] Stopping as Success. (n.d.). About Us. Retrieved June 2020 from https://www.stoppingassuccess.org/about-us

[16] Stopping as Success. (2020, March 17). USAID Mission Checklist for Sustainable Transitions. Retrieved from https://www.stoppingassuccess.org/resources/usaid-mission-checklist/

[17] European Commission. (2015, December). EU Site Explorer: Overview. Retrieved June 2020 from https://euaidexplorer.ec.europa.eu/content/overview_en

[18] Froidure, P., & Devillé, H. (2015). Review of the risks related to a results oriented approach for EU development and cooperation action. Retrieved from https://op.europa.eu/en/publication-detail/-/publication/b2921ec1-c589-11e5-a4b5-01aa75ed71a1

[19] European Commission Directorate-General for International Cooperation and Development. (2018, November 19). The New European Consensus on Development: ‘Our World, Our Dignity, Our Future’. Retrieved from https://op.europa.eu/en/publication-detail/-/publication/5a95e892-ec76-11e8-b690-01aa75ed71a1

[20] European Commission. (2018, June). Joint Programming Guidance: Supporting EU Delegations to Work Better Together with Member States, Like-Minded Partners and Country Stakeholders. Retrieved from https://knowledge.effectivecooperation.org/node/77

[21] Parry, M., & Segantini, E. (2017, October). How the EU Budget is Spent: Development Cooperation Instrument. Retrieved from https://www.europarl.europa.eu/RegData/etudes/BRIE/2017/608764/EPRS_BRI(2017)608764_EN.pdf

[22] Linköping University. (2020, May 25). Reporting to the European Commission. Retrieved June 2020 from https://insidan.liu.se/forskningsfinansiering/horizon-2020/rapportera-till-eu-kommissionen?l=en

[23] European Commission: Intelligent Energy Europe. (2012). Guidelines for Drafting the Final Technical Implementation Report. Retrieved June 2020 from https://ec.europa.eu/easme/en/technical-reporting

[24] LIFE Program. (2019, June 3). 2019 Operating Grants for NGOs. European Commission. Retrieved from https://ec.europa.eu/easme/en/section/life/2019-life-call-proposals-ngos

[25] Wright, H., Dimsdale, T., & Healy, C., et al. (2018, June 26). Sustainable Infrastructure and the Multilateral Development Banks: Changing the Narrative. E3G. https://www.e3g.org/library/sustainable-infrastructure-and-the-multilateral-development-banks-changing

[26] Naidoo, C., Amin, A., & Dimsdale, T., et al. (2014, April 9). Strategic national approaches to climate finance: Report on scoping work in Peru, Chile and Colombia on national climate finance pathways and strategies. Retrieved from https://www.e3g.org/news/media-room/strategic-national-approaches-to-climate-finance

[27] LIFE Program. (2019, June 3). 2019 Operating Grants for NGOs. European Commission. https://ec.europa.eu/easme/en/section/life/2019-life-call-proposals-ngos

[28] Carbon Market Watch. (2014, June 25). The Joint Implementation offsetting mechanism is deeply flawed – time to consign it to history? Retrieved from https://carbonmarketwatch.org/2014/06/25/the-joint-implementation-offsetting-mechanism-deeply-flawed-time-to-consign-it-to-history/

[29] Carbon Market Watch. (n.d.) CDM Project Cycle. Retrieved 2019, May 10 from
https://web.archive.org/web/20190510064748/http:/archive.carbonmarketwatch.org/learn-about-carbon-markets/cdm-project-cycle/

[30] United Nations Framework Convention on Climate Change. (n.d.). Guidelines for Completing the Project Design Document (CDM-PDD), the Proposed New Methodology: Baseline (CDM-NMB) and the Proposed New Methodology: Monitoring (CDM-NMM). Retrieved June 2020 from https://cdm.unfccc.int/Reference/Documents/Guidel_Pdd/English/Guidelines_CDMPDD_NMB_NMM.pdf

[31] Directorate-General for European Civil Protection and Humanitarian Aid Operations. (n.d.). Agreements for Humanitarian Aid and Emergency Support Within the EU Awarded in 2017 by DG ECHO. Retrieved June 2020 from https://ec.europa.eu/echo/files/funding/agreements/agreements_2017.pdf

[32] People in Need Ethiopia. (n.d.). Final Evaluation: PIN’s Support to Strengthening Agricultural Extension Services in SNNPR, Ethiopia. Retrieved from https://www.clovekvtisni.cz/en/learning-series-on-agricultural-extension-738pub

 

 

Sustaining “Sustainable Development”

Posted by on Apr 20, 2020 in Aid effectiveness, climate change, environment, ex-post evaluation, Impact, international development, JICA, OECD, Participants, post project evaluation, Sustainability, Sustainable development, USAID, Vietnam, water/ sanitation | 0 comments

Sustaining “Sustainable Development”

 

Sustaining “Sustainable Development”?

 

As a global development industry, we have almost no evidence of how (un)sustained the outcomes or impacts of 99% of our projects because we have never returned to evaluate them. But from early indications based on the ex-posts, we have evaluated 2-20 years after donor departure it is, learning from what was and was not sustained is vital before replication and assuming sustainability. Most results taper off quite quickly, showing 20-80% decreases as early as two years post-closure and donor exit. A few cases of good news also appear, but more trajectories falter and fail than rise or remain. Sustainability, then, is not a yes-no answer, but a how much, yet too few ask… hence if they were, resilient, they are less so, or even not at all, now.

 

At Valuing Voices we focus on the sustainability of projects after external support ends. Still, those projects are also dependent on the viability of the environment in which they are based. As Andy Rowe, an evaluator on the GEF’s Adaptation Fund board, noted at IDEAS’ Conference in Prague late 2019 [1], a need for sustainability-ready evaluation to help us know how viable the resources are on which so many of our projects rest [2]. He states, “the evaluation we have today treats human and natural systems as unconnected and rarely considers the natural system”. He goes on to differentiate between biotic natural capital  (air, water, plants, and trees) and abiotic natural capital sources (fossil fuels, minerals, and metals, wind, and solar).

 

How much are projects designed assuming those resources are and will remain plentiful? How often do we evaluate how much our projects drain or rely on these environmental elements? Many projects are required to do environmental compliance and safeguarding against damage at project onset [3]. Others, such as agriculture and natural resource management or water/ sanitation, often focus on improving the environment on which those activities rely, e.g., improving soil or terrain (e.g., terraces, zais), planting seedlings, and improving access to potable water for humans and animals. Still, many projects ‘assume’ inputs like rainfall, tree cover, solar power, or do not consider the sustainability of natural resources for the communities in which they intervene. Examples are both those that rely on natural systems as well as those supposedly beyond them, e.g., enterprise development, education, safety nets, etc. Yet many enterprises, schools, safety nets do rely on a. viable environment in which their participants trade, learn, and live, and all are subject to the growing climate change disruptions. 

 

Why is this urgent? The OECD/DAC reminds us that “Natural assets represent, on average 26% of the wealth of developing countries compared to 2% in OECD economies” [4]. Unless we protect them and address the demand for natural resources, demand will far outstrip supply. “By 2030, an additional 1 billion people are expected to live in severely water-stressed areas, and global terrestrial biodiversity is expected to decline an additional 10%, leading to a loss of essential ecosystem services. By 2050, growing levels of dangerous air emissions from transport and industry will increase the global number of premature deaths linked to airborne particulate matter to 3.6 million people a year, more than doubling today’s levels. Failure to act could also lead to a 50% increase in global greenhouse gas emissions by 2050, and global mean temperature increases of 3-6°C by the end of the century, in turn contributing to more severe and sometimes more frequent natural disasters… [so] reconciling development with environmental protection and sustainable resource management is broadly agreed as a central concern for the post-2015 development agenda.”

 

When we return to projects that are a mix of behavior change and environment, we find a wide range of results:

  • Some projects, such as JICA Vietnam’s water supply and irrigation infrastructure reached 80% of the final results two years later [5]. And while the pilot projects were worse off (as low as 28% of irrigated hectares), longer-standing projects sustained as much as 72% of final results. While such agricultural development assumes continued water supply and access, does it evaluate it? No.
  • Some can define what ex-post lessons are more narrowly as functioning mechanisms: New ex-posts of water/ sanitation showed better – but still mixed results, such as USAID Senegal’s [6]. “While a majority (63 percent) of the water points remained functional, the performance varied significantly based on the technology used. Of the different technologies, the Erobon rope pumps performed poorly (27 percent functional), while the India Mark (74 percent functional) and mechanized pumps (70 percent functional) performed the best.”
  • Some projects that include environmental considerations illustrate our point by only focusing on behavior change as this sanitation/ hygiene ex-post from Madagascar did, where results fell off precipitously three years ex-post but without considering water supply or quality much [7]. 

[7]

  • There can be useful learning when one combines an evaluation of both types of sustainability (ex-post and environmental). A JICA irrigation project in Cambodia shows that when irrigation canals were mostly sustained over the five-years ex-post, they could serve increasing needs for land coverage and rice production [7]. The area of irrigated fields at the national level in 2010 reached the target, and the irrigated field area has since continued to increase in most areas. Even the largest drop [in area irrigated] post-closure was only 11%. They reported that the unit yield of rice at the end-line survey in 2012 at 11 sites was 3.24t/ha (average) versus 3.11t/ha of unit yield of rice at the ex-post evaluation in 2017, which [almost] maintains the 2012 level. The ex-post showed that “continuous irrigation development in the said site can be considered as the main reason for the increase in land area. Securing an adequate amount of water is an important factor in continuously improving rice productivity.” The research also found that 81% of agricultural incomes as a result of the irrigation had increased, 11% stayed the same, and 8% had decreased. Again, this looks to be among the most resilient projects that, based on ex-post research, included environment which was also found to be as resilient as the livelihoods it was fostering.
  • Sometimes more bad than good news is important when tracking environment and ex-post sustainability: Food for the Hungry, ADRA, and CARE Kenya found that unreliable water supply reduced the motivation to pay for water, threatening the resources to maintain the system [8]. What improved prospects of sustainability understand why communities could not sustain water and sanitation results based on willingness-to-pay models, as well as water being unavailable. Further, a lesson the organizations ideally learned was that “gradual exit, with the opportunity for project participants to operate independently prior to project closure, made it more likely that activities would be continued without project support.” So the question remains, what was learned by these organizations to avoid similar bad results and improve good, resilient results in similar circumstances?

 

[6]

 

Neither sustainability nor environmental quality can be assumed to continue nor to have positive results. Both are extensively under-evaluated, and given climate change disruptions, and this must change. Rowe concludes: “Climate change is a major threat to the long-term sustainability both attacking the natural systems (e.g. lower rainfall or higher floods, worse soil quality, increasing pests attacking crops, disappearing fish stocks, microplastics in our air and water, increasing sea levels from melting glaciers, worsening public health etc.) and destabilizining our Earth’s regenerative capacity. Fortunately, technical barriers do not prevent us from starting to infuse sustainability into evaluation; the barriers are social and associated with the worldview and vision of evaluation.”

 

Sources:

[1] IDEAS 2019 Global Assembly. (n.d.). Retrieved from https://2019.global-assembly.org/

[2] Rowe, A. (2019). Sustainability‐Ready Evaluation: A Call to Action. New Directions for Evaluation, 162, 29-48. Retrieved from https://www.researchgate.net/publication/333616139_Sustainability-Ready_Evaluation_A_Call_to_Action

[3] USAID. (2013, October 31). Environmental Compliance Procedures. Retrieved from https://www.usaid.gov/our_work/environment/compliance/pdf/216

[4] OECD. (2015). Element 4, Paper 1: Global and local environmental sustainability, development and growth. Retrieved from https://www.oecd.org/dac/environment-development/FINAL%20POST-2015%20global%20and%20local%20environmental%20sustainability.pdf

[5] Haraguchi, T. (2017). Socialist Republic of Viet Nam: FY 2017 Ex-Post Evaluation of Japanese ODA Loan Project “Small-Scale Pro Poor Infrastructure Development Project (III)”. Retrieved from https://www2.jica.go.jp/en/evaluation/pdf/2017_VNXVII-5_4.pdf

[6] Coates, J., Kegode, E., Galante, T., & Blau, A. (2016, February). Sustaining Development: Results from a Study of Sustainability and Exit Strategies among Development Food Assistance Projects: Kenya Country Study. USAID. Retrieved from https://www.globalwaters.org/resources/assets/ex-post-evaluation-senegal-pepam

[7] Madagascar Rural Access To New Opportunities For Health And Prosperity (RANO-HP) Ex-Post Evaluation. (2017, June 1). USAID. Retrieved from https://www.globalwaters.org/resources/assets/madagascar-rural-access-new-opportunities-health-and-prosperity-rano-hp-ex-post-0

[8] Kobayashi, N. (2017). Kingdom of Cambodia: FY2017 Ex-Post Evaluation of Technical Cooperation Project: “Technical Service Center for Irrigation System Project – Phase 2 / The Improvement of Agricultural River Basin Management and Development Project (TSC3)”. Retrieved from https://www2.jica.go.jp/en/evaluation/pdf/2017_0900388_4.pdf

 

Wishing for ex-post evaluation Christmas Lights rather than Needles in Haystacks

Posted by on Dec 11, 2019 in capacity, ex-post evaluation, Failure, Indonesia, JICA, Monitoring and Evaluation, RSA South Africa, water, World Vision | 5 comments

 

Wishing for Ex-Post Evaluation Christmas Lights
Rather than Needles in Haystacks

 

This is what life of most ex-post evaluation researchers looks like, mostly without the counting congratulator:

 

I recently spent three days looking for ex-post evaluations for a client across nearly a dozen organizations. I was hard-pressed to find 16 actual ones. Sorting through ‘impact evaluations’ that were done in the middle of implementation does not tell us anything about what was sustained after we leave, nor do delayed final evaluations that happen to be done after closure. While these (rightly) focus on cost-effectiveness, relevance and efficiency, measures of sustained impact are projections, not actual measures of what outcomes and impacts stood the test of time. I weeded out some desk studies that did not return to ask anyone who participated. Others titled ‘ex-post’ were barely midterms (I can only gather they misconstrued ‘ex-post’ as after-starting implementation?) and a few more reports only recommended doing ex-post evaluation after this final evaluation. For more lessons on how random and misconstrued ex-posts can be, see Valuing Voices’ research for Scriven. None of these 16 actual ex-posts even told us anything about what emerged (as we look at during Sustained and Emerging Impacts Evaluations) from local efforts in the years after assistance ended [1].

This is what I wish my ex-post haystack would look like, bountiful treasures of numerous ex-post-project evaluations, as numerous as these Christmas lights here in Tabor, Czech Republic.

 

If we had more ex-posts to learn from, we could learn from what lasted. What could locals sustain? Why? Why not? How can we do better next time? We could compare across sectors and countries, and we could see what conditions and processes during implementation supported sustainability -and importantly – why some failed, so we don’t repeat those mistakes.

We could move from our current orange slices that ends at closure to green sustainability of the project cycle:

I will be adding the ones I found to our Catalysts list soon, but when my client asked me who held databases of ex-post evaluations, I had to say only Valuing Voices and Japan’s JICA (since 1993 who even differentiates the ex-posts between Technical Grants and ODA Loans). This is not to say some cannot be found by trawling the OECD or the World Bank, but this is Needle-in-Haystack work again and so there are only 2 databases to learn from. Isn’t that shocking?

Now JICA has really upped the illumination ante, so to speak: They are now doing what they are calling JICA’s Ex-post Monitoring’ which was like Christmas come early [2]! Returning to learn 7 years after the ex-post which was 1-3 years after closure meant one could evaluate the sustained impacts of results, see if JICA’s recommendations to their partners had been implemented, how they had adapted to changes over a decade post-closure and find learning for new programming. While it was unclear why these specific projects were selected, it is amazing they are doing 5-10 per year. They are my ex-post gods/ goddesses and I fawned over two JICA evaluators at the last European Evaluation Society Conference.

“Ex-post monitoring is undertaken 7 years after a project was completed in principle in order to determine whether or not the expected effects and impacts continue to be generated, to check that there are no sustainability-related problems with the technical capacities, systems and finances of the executing agency nor with the operation and management of developed facilities, etc., and to ascertain what action has been taken vis-a-vis the lessons learned and recommendations gleaned during the ex-post evaluation.”

 

What we can learn from returning again is illustrated by one of JICA’s water project loans in RSA, which ended in 2003, had an ex-post in 2006, followed by monitoring of sustainability in 2013 [3]. While the report included issues of data access and evaluators expressed caution in attributing causation of positive changes to the project, but it not only continued functioning, the government of South Africa (RSA) solved barriers found at the ex-post:

  • “Data for the supply and demand of water pertaining to the Kwandebele region could not be obtained. However, considering the calculation from the water supplied population and supplied volume and the result from the DWAF interview, water shortage could not be detected in the four municipalities studied by this project…” [3]
  • “The ex-post evaluation indicated that the four components were not in the state to be operated and managed effectively. Currently, the components are operated and managed effectively and are operating under good condition [and] concerning sustainability, improvement can be seen from the time of ex-post evaluation. Shortage of employees and insufficient technical knowledge has been resolved…” [3]
  • “Compared to the time of ex-post evaluation, improvement was seen in the under-five mortality and life expectancy. However, since the components implemented by this project are limited in comparison with the scope of the project, it is impossible to present a clear causal relationship” [3].

In another, from Indonesia’s air quality testing labs which involved capacity building and equipment maintenance 6 years after the ex-post, they mostly found training and use continued despite organizational changes and maintenance challenges: [4]

  • “After the ex-post evaluation, many of the target laboratories changed their affiliation from the Ministry of Public Works (MOPW) and MOH to provincial governments. While the relocation of equipment has been carried out in a handful of provinces, in other provinces equipment is still located at the laboratories where it was originally installed and these laboratories still have the right of use” [4]
  • In spite of some irregularities ”As the Ministry of Environment (MOE) still has ownership of the equipment, some laboratories have inappropriate audit results that show allocation of O&M budget to equipment which is not included in their accounting…” [4]
  • “Out of 20 laboratories where the questionnaire survey confirmed that equipment still remained, 15 laboratories replied that spare parts for equipment are still available but are difficult to obtain…It takes several months to one year to obtain spare parts, occasionally out of Indonesia, even if a repair service is available” [4].

In this case, there were lessons learned for JICA and Indonesia’s Ministry of the Environment programs about ownership and the right use of the equipment and retiring obsolete equipment. Talk about a commitment to learning from the ongoing success or failure of one’s projects!

 

As you have read here on Valuing Voices for more than six years, unless we include post-project sustainability that asks our participants and partners how sustained their lives and livelihoods could be, and even resilient to shocks like political or climate change, we cannot say we are doing Sustainable Development. We need such lessons about what could be sustained and why.

We can prepare better to foster sustainability. In the coming months we are working on checklists to consider during funding, design, implementation, M&E pre-and post-exit, to foster sustainability. Will keep you posted, but as World Vision also found: “Measuring sustainability through ex-posts requires setting clear benchmarks to measure success prior to program closure, including timelines for expected sustainment.”

And as my gift to you this Holiday Season, let me share WV’s Learning Brief about Sustainability, with wise and provocative questions to ponder about dynamic systems, benchmarking, continuous learning, attribution, and managing expectations [5]. World Vision shares how infrastructure and community groups and social cohesion fared well, yet lessons circled back to the need for JICA-like ‘monitoring’ and mirror rich ex-post lessons from FFP/Tufts (Rogers, Coates) and Hiller et al. that explains why we do ex-posts at all: “Project impact at the time of exit does not consistently predict sustainability“ [6].

 

Now my gift: a few big lessons from  the six years of researching sustainability across the development spectrum.  I have found no evaluations that were only positive. Most results trended downwards, a few held steady, and all were mixed. We cannot assume the sustainability of results at closure, nor optimistic projections as we’ve seen in the climate arena.

Please consider:

  • Designing with our participants and partners so what we do,
  • Implementing with partners far longer to make sure things still work,
  • Adapting exit based on benchmarks to see how well the resources, partnerships, capacities, and ownership have been transferred,
  • Using control or comparison groups to make sure ‘success’ was due to you and being careful about attributing results to your projects while considering how you contributed to a larger whole of ongoing country progress or stagnation,
  • Being willing to jettison what is unlikely to be sustained and learn from what we designed and implemented poorly (due to our design, their implementation, external conditions),
  • Given climate-change, learning fast, adaptively and revising fast given changing conditions,
  • Without knowing what has been sustained we cannot replicate nor scale-up,
  • Sharing lessons with your leaders – for people’s lives depend on our work,
  • Learning from what emerged as our participants and partners refashioned implementation in new ways could sustain it (without the millions we brought),
  • Refocusing ‘success’ from how much we have spent, to how much was sustained.

 

Please make our next Christmas merry. Do MANY ex-post evaluations, Learn TONS, Share WIDELY WHAT WORKED AND FAILED TO WORK (you will be praised!), and let’s CHANGE HOW WE DO SUSTAINABLE DEVELOPMENT.

 

May 2020 bring health, happiness, and to all of us a more sustainable world!

 

 

Sources:

[1] Cekan, J., Zivetz, L., & Rogers, P. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE

[2] JICA. (n.d.). Ex-post Monitoring. Retrieved December, 2019, from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/index.html

[3] Matsuyama, K. (2012). Ex-Post Monitoring of Japanese ODA Loan Project: South Africa, Kwandebele Region Water Augmentation Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/2012_full_03.pdf

[4] Kobayashi, N. (2009, August). Ex-post Monitoring of Completed ODA Loan Project: Indonesia, The Bepedal Regional Monitoring Capacity Development Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/indonesia2008_01.pdf

[5] Trandafili, H. (2019). Learning Brief: What does sustainability look like post-program? Retrieved from https://valuingvoices.com/wp-content/uploads/2019/12/Sustainability-Learning-Brief_final_WV-icons.pdf

[6] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fsnnetwork.org/ffp-sustainability-and-exit-strategies-study-synthesis-report

 

Learning from a river of ex-post project evaluations, tools and guidance… Thanks USAID!

Posted by on Nov 10, 2019 in Aid effectiveness, civil society, ex-post evaluation, Ghana, Impact, impact evaluation, Indonesia, Maternal Child Health, MCHN, NGOs, post-project evaluation, sanitation & hygiene, Senegal, Sustainable development, USAID, water/ sanitation | 1 comment

Learning from a river of ex-post project evaluations and tools… Thanks USAID!

Dear ex-post aficionados. It’s raining ex-post project evaluations. Here’s hoping learning from such evaluations in water/ sanitation, maternal/child health and even capacity building/ peacekeeping, and their number increases!

 

1. WATER/ SANITATION & HYGIENE:

USAID has a series of six ex-post evaluations of the water/ sanitation and hygiene sectors since 2017! What is exciting is that they are also looking to the future. These evaluations will “provide insight into what happens after an activity ends, and how to mitigate challenges in future programming, potentially. The series will inform USAID’s WASH activity design and implementation and contribute to a larger sector discussion on achieving sustainability.”

The E3 water division (Water CKM ) took sustainability on as their strategy and have made great strides these last two years. They have done five ex-post project evaluations, cited below, and MSI has completed one more wat/san/ hygiene ex-post evaluations, specifically:

Madagascar Rural Access to New Opportunities for Health and Prosperity (RANO-HP) – Published June 2017
The first evaluation in the series explores the sustainability of the sanitation and hygiene components of the RANO-HP activity, implemented in 26 communes from 2009–2013.

Indonesia Environmental Services Program (ESP) – Published August 2017
The second evaluation in the series examines the sustainability of water utility capacity building, microcredit, and financial outcomes associated with the ESP activity, which was implemented from 2004–2010.

Ethiopia Millennium Water Alliance (MWA-EP) – Published May 2018
The third evaluation in the series examines the long-term sustainability of outcomes related to rural water point construction, rehabilitation, and management, as well as participatory sanitation and hygiene education and construction related to the MWA-EP activity, implemented in 24 rural districts between 2004–2009.

Financial Institutions Reform and Expansion–Debt and Infrastructure (FIRE-D) – Published September 2018
This evaluation is the fourth in the series. It examines how urban water and sanitation services in India have changed since FIRE-D closed and to what extent policies, practices, and financing mechanisms introduced through FIRE-D have been sustained.

Millennium Water and Sanitation Program in Senegal (PEPAM/USAID) – Published July 2019
The fifth ex-post evaluation in the series looks at the PEPAM project (Programme d’Eau Potable et d’Assainissement du Millénaire au Sénégal), implemented from 2009–2014 to improve sustainable access to WASH in four regions of Senegal.

 

USAID-funded by MSI: USAID/Ghana’s Water Access, Sanitation, and Hygiene for Urban Poor (WASH-UP)– published Nov 2018

Also USAID and Rotary International developed a WASH Sustainability Index Tool, “to assess a WASH activity’s likelihood to be sustainable according to the following factors: availability of finance for sanitation; local capacity for construction and maintenance of latrines; the influence of social norms; and governance.” This is similar to what we learned from USAID/ FFP/ Tufts/ FHI360 12 ex-posts that resources, capacities, motivation and linkages (aka partnerships, including governance) are vital to sustaining outcomes and impacts.

 

It will be interesting to see whether they examine the other ex-posts for excellent lessons, as they have the Senegalese evaluation:

  • “Whether or not to subsidize sanitation access …Based on this evaluation’s findings and exploration of the literature, subsidies can help improve the quality of household latrines, but increasing use of those latrines remains a challenge.
  • In contrast, CLTS (a nonsubsidized approach) is often credited with increasing use of unimproved latrines, but serious questions linger about quality and long-term sustainability of the latrines built after CLTS triggering, particularly as it relates to moving up the sanitation ladder. This evaluation… provides the opportunity to examine the potential value of a hybrid approach….
  • The handwashing results suggest that low-cost, low-quality handwashing stations such as tippy taps do not lead to sustained behavior change. It may be worth considering hygiene investments that reduce the behavior change burden on targeted beneficiaries.

2. MATERNAL/ CHILD HEALTH & NUTRITION:

 “Sustainability of a Community-Based CHOICE Program to Improve the Health and Nutrition Status of Mothers and Infants in Indonesia,” The report focused on whether the USAID-funded CHOICE program had left sustainable impacts: improving the health and nutrition status of children under the age of five, as well as the health status of pregnant and lactating women and mothers or caretakers of young children in the Pandeglang District of Indonesia. “After examining the data collected from the PSS, the researchers found that there were significant improvements in many indicators—such as births attended by skilled personnel, the treatment of diarrhea, and the nutritional quality of food fed to infants—in the six years after the CHOICE program ended. However, despite these improvements, the researchers found no significant statistical differences between villages that received the CHOICE program interventions and comparison villages, which did not. This speaks to using such a comparison methodology to focus on actual contribution and rule out the “rising tide lifts all boats” phenomenon.

 

3. CAPACITY DEVELOPMENT & PEACEBUILDING:

USAID’s Regional Office in Thailand evaluated its capacity building and peacebuilding program 1.5 years ex-post.  While civil society was strengthened and there were inroads made on peacebuilding,many interventions initiated during Sapan did not continue post-Sapan, although some did remain. For example, “stakeholders cite evidence of continuing to use some governance tools in local governance related to service delivery [although] because of limited financial resources after Sapan ended, they had to change some of their interventions and reduce the range of people they could include. There are lessons for whose capacities are built, two-way feedback loops with local partners, using local organizations such as universities to sustain training, planning sufficient time for partners to internalize training lessons, etc.

 

4. USAID FUNDED GUIDANCE:

‘Impact Evaluations’ have a new focus on long-term impact, rather than effectiveness during implementation (which was at least the original intent of impact evaluation in the 1980s)! In September 2018, USAID and Notre Dame issued a Guide for Planning Long-term Impact Evaluations as part of the Utilizing the Expertise of the ERIE Program Consortium. The guide covers the difference between traditional impact evaluation designs and data collection methods and how to apply them to long-term impact evaluations (LTIE). It also shares examples across a range of sectors, including later evaluating past impact evaluations, which ended before final evaluation.

Finally, in new 2018 USAID guidance, ex-post evaluation is clarified as the source of the sustainability of services and benefits. USAID clarifies that “questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation, a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be, and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project’s services and benefits were sustained.”

Such richness that we can learn from. Keep the momentum going on the 99% of all global projects yet unevaluated ex-post, and change how we fund, design, implement, monitor and evaluate global development projects!