Blog

“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School

Posted by on Apr 9, 2021 in business, environment, ESG, ex-post evaluation, foreign aid, Impact, impact investing, OECD, Sustainable development | 4 comments

“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School

Recently I’ve had conversations where I had to define which sustainability we were talking about. Was it:

  1. ex-post-project sustainability of outcomes and impacts,
  2. environmental sustainability, or
  3. business sustainability?

 

Since I spend most of my time evaluating the ex-post sustained and emerging impacts of foreign aid projects years after projects close, or at least advocate for it, let’s start there.

  1. The Organisation for Economic Co-operation and Development (OECD) is a “forum and knowledge hub for data and analysis, exchange of experiences, best-practice sharing, and advice on public policies and international standard-setting.” Regarding evaluation specifically, the OECD has “established common definitions for six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact, and sustainability – to support consistent, high-quality evaluation”. Focusing on long-term sustainability, their evaluation guidance is:

 

 

Source: OECD, Better Criteria for Better Evaluation, 2019

 

The good news is that in this recent publication on Applying Evaluation Criteria Thoughtfully (2021), OECD keeps the updated definition but inches towards recommending actual ex-post project sustainability evaluation, rather than just projected (and assumed “likely to continue” sustainability). For this, “likely” is the most significant reason evaluators for donors and implementers have assumed, rather than evaluated, sustainability for decades. Further, positive, ‘sustained’ trajectories are also assumed at close-out/ exit, but rarely tested ex-post.

The OECD criteria give not evaluating it as an option. I far prefer “net benefits of the intervention continue” as it is a marching order: Prove that results were sustained. In this evolution, this 2021 report states, “After the completion of the intervention, and evaluation of sustainability would look at whether or not the benefits did continue, this time drawing on data and evidence from the intervention’s actual achieved benefits.”

 

OECD even goes on to recommend implementing and monitoring for sustainability. The new piece de resistance is: “Sustainability should be considered at each point of the results chain and the project cycle of an intervention”:

  1. “The sustainability of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in the broader context of the intervention…. as well as whether there was willingness and capacity to sustain financing (resources) at the end of the intervention
  2. For example, an evaluation could assess whether an intervention considered partner capacities
  3. Built ownership at the beginning of the implementation period…. And
  4. In general, evaluators can examine the conditions for sustainability that were or were not created in the design of the intervention and by the intervention activities and whether there was adaptation where required.”

Yes, Valuing Voices highlighted this at the American Evaluation Association presentation “Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation” in 2016, and we have developed this into 2020’s Exiting for Sustainability trainings and checklists. Wonderful to see implementing for sustainability in guidance by the OECD!

 

Moreover, while the 2019 OECD report mentioned resilience in passing, related to sustainability, “encourages analysis of potential trade-offs, and of the resilience of capacities/ systems underlying the continuation of benefits”. Such resilience and continuation of benefits evaluation involve examining huge systems (the financial, economic, social, environmental, and institutional capacities) that projects and programs are implemented within, whose stability is needed to sustain net benefits over time. Yes, for ex-post sustainability questions for evaluators to consider should include: “To what extent did the intervention contribute to strengthening the resilience of particularly disadvantaged or vulnerable groups” on which the sustained impacts of so much of our “Leave No One Behind” myth of Sustainable Development rely.

 

However, OECD makes suggestions to evaluate even broader, overwhelming what is feasible: “…this involves analyses of resilience, risks, and potential trade-offs.” Whose? All stakeholders, from participants to local partners and national and international implementers, and international donors? How far back and how far forward? What a huge undertaking. Further, the OECD points evaluators to define resilience, but as I learned in my Famine Early Warning System research and a current ex-post evaluation process for the Adaptation Fund, that involves creating evaluable boundaries by determining resilient to what kinds of shocks? Vital questions current industry monitoring and evaluation budgets for all evaluations, much less (too-rare) ex-post project evaluations, are insufficient for as they hover around 3-5% of total costs.

 

  1. Slight progress at OECD is being made by acknowledging environmental sustainability first brought up by the Brundtland Report, “Our Common Future” back in 1987. This linchpin report highlighted that “critical global environmental problems were primarily the result of the enormous poverty of the South and the non-sustainable patterns of consumption and production in the North. It called for a strategy that united development and the environment – described by the now-common term “sustainable development”… that meets the needs of the present without compromising the ability of future generations to meet their own needs.”

 

While an OECD brief in 2008 considers the environmental aspects of our thinking about sustainability, it argues that sustainability primarily about “using economic development to foster a fairer society while respecting ecosystems and natural resources.” The 2021 Applying Evaluation Criteria Thoughtfully rather unhelpfully mostly ignores the environment’s role in sustainability: “Confusion can arise between sustainability in the sense of the continuation of results, and environmental sustainability or the use of resources for future generations. While environmental sustainability is a concern (and maybe examined under several criteria, including relevance, coherence, impact, and sustainability), the primary meaning of the criteria is not about environmental sustainability as such; when describing sustainability, evaluators should be clear on how they are interpreting the criterion.” Given rapid climate change, I would argue that any sustained and emerging outcomes and impacts of projects that does not include an evaluation of the environmental context will fail to foster sustained resilience. Yet donors’ fixed funding timeframes that set completion to disbursement without evaluating sustainability or resilience continue to be huge barriers.

 

  1. Finally, business sustainability brings together these impacts on communities and society along with impacts on the environment. These are called ESG (Environmental, Social, and Governance) criteria. A Harvard Business School brief defines sustainability as “doing business without negatively impacting the environment, community, or society as a whole. “Where applied well, the aspiration is that “beyond helping curb global challenges, sustainability can drive business success.” While Harvard Business Review highlights “What Works’ in Calculating the Value of Impact Investing, they are, like almost all of global development ‘while- we-are-there’-measures. There is one mention of ‘terminal value’, 5 years after close of ownership, and they estimate social return on investments. This is a good, step, but as insufficient as foreign aid – for these are projected, not actual results.

At Valuing Voices, we have found hopeful examples such as IKEA as well as where ‘impact investing’ hype does not match the claims. Nonetheless, increasingly businesses are trying to consider circular economy systemic principles of “economic development designed to benefit businesses, society, and the environment.” This is regenerative, aims to decouple growth from the consumption of finite resources, not generate excess waste that cannot be reused and actuals seem to be measured at least during investments. As Harvard notes, “this leads investors to look at factors such as a company’s carbon footprint, water usage (both Environment), community development efforts (Social), and board diversity (Governance).” We encourage them to measure long-term/ longitudinally.

Australia’s RMIT defines business sustainability as comprising 4 pillars: Human, Social, Economic, and Environmental which combines a) “Human sustainability focuses on the importance of anyone directly or indirectly involved in the making of products, or provision of services or broader stakeholders;… b) Social sustainability focuses on maintaining and improving social quality with concepts such as cohesion, reciprocity and honesty and the importance of relationships amongst people;… c) Economic sustainability aims to improve the standard of living [and] the efficient use of assets to maintain company profitability over time;… d) Environmental sustainability places emphasis on how business can achieve positive economic outcomes without doing any harm, in the short- or long-term, to the environment.”

Would ESG success be sustained over the long-term rather than short-term shareholder profit cycles? Will the OECD start to recommend extensive ex-post evaluation? Will they develop guidance to incorporate environmental concerns in evaluation for our common good? I do not yet know, but I implore these silos to start talking. No time to waste!

As my colleague and  collaborator Susan Legro commented, we need to:

1) Continue to seek clarity and specificity in the terminology that we use, ensuring that it is clear to all stakeholders and beneficiaries; and

2) Find ways to study projects and initiatives over the longer term, which is the only way to study the designation of “sustainable” for any initiatives seeking that label.

What are your thoughts?

 

Reblog USAID’s Unpacking the Drivers of WASH Sustainability

Posted by on Mar 22, 2021 in ex-post evaluation, USAID, water/ sanitation | 0 comments

Reblog Unpacking the Drivers of WASH Sustainability

In Nov 2019 I highlighted several water/ sanitation/ hygiene ex-posts happening at USAID’s E3 Water Bureau. They’ve just come up with a synthesis, which is important, both in learning and transparency for accountability. Kudos! My only quibble is that we don’t know the full scale after only 6 very different ex-posts.

The authors Jones & Jordan’s overview: In its evaluation series, USAID looked back at the results of six water, sanitation, and hygiene (WASH) activities to inform future USAID investments in the sector and to better understand the long-term impact and sustainability of its interventions several years after projects close. Here: https://www.globalwaters.org/resources/blogs/unpacking-drivers-wash-sustainability

The water, sanitation, and hygiene (WASH) Ex-Post Evaluation Series represents a key milestone in the Agency’s ongoing quest to unpack the drivers of sustainability within our WASH programs. The commitment, made in the first USAID Water & Development Strategy (2013–2018), came in the wake of headlines about high rates of nonfunctional water systems in partner countries where USAID has long invested. At that time, many governments and development partners were grasping for similar answers, and a number of sustainability assessment tools emerged. For its part, the Agency pursued a multi-pronged approach that focused on: 1) measuring progress towards sustainable outcomes (such as through the development and application of the Sustainability Index Tool with Rotary International); 2) gaining a better understanding of the drivers of long-term outcomes through this WASH Ex-Post Evaluation Series; and 3) testing new ideas, approaches, and tools to strengthen the local systems that can deliver WASH service sustainability through the Sustainable WASH Systems (SWS) Learning Partnership.

The Agency, together with its development partners, has achieved staggering results in terms of delivering first-time water and sanitation access to people the world over. Since 2013, USAID has helped roughly 25 million people gain access to at least basic drinking water services and 18 million people gain access to at least basic sanitation. Together with partners, USAID has mobilized almost $100 million in new funding for the sector and supported nearly 17,000 communities become open defecation free.

Yet the results of this ex-post series are sobering. Despite tremendous achievements within the life of our programs, they have largely not endured. This is especially the case in countries and communities with the highest levels of poverty at baseline, where the Agency’s resources are needed the most. Rural water systems that, at activity close, delivered safe water to households have fallen into disrepair. Basic latrine ownership and use have dwindled. Communities certified as open-defecation free are backsliding, and gains in handwashing have not been sustained.

The series did reveal some programming bright spots. Where USAID invested in providing technical assistance to committed government partners and utilities, gains in service provision and local capacity were sustained, with local actors taking up and expanding upon best practices introduced during activity implementation. Often these successes endured in countries and communities that had higher levels of capacity at the outset. However, the successes in these contexts demonstrate important lessons about investing time and resources into partnering with local institutions and focusing on plans for management of services, not just first time access.

In the course of the roughly 15 intervening years since most of the activities evaluated in this series were designed, the sector has evolved. For instance, in resource-constrained environments, the sector is now coalescing around facilitating the development of professionalized support to community-managed rural water schemes in various forms, rather than expecting voluntary committees to manage essential services alone. Additional approaches beyond community-led total sanitation, including smart subsidies and market-based sanitation, are seen as necessary to move households up the sanitation ladder, and are being applied through USAID programming. And the Agency is shifting its WASH social and behavior change programming to more holistic approaches that address emotional drivers, convenience, and social norms to modify intractable behaviors rather than communication or health promotion alone. The WASH Ex-Post Evaluation Series validates why those shifts were essential.

All those with a stake in promoting lasting development gains in the sector need to internalize these findings and take a long look in the mirror. USAID has and is seeking to do better. First, under the USAID Water and Development Plan (2018–2022) in support of the Global Water Strategy, USAID codified its commitment to sustainability with the goal of increasing the availability and sustainable management of safe water and sanitation, and an emphasis on improving the underlying governance, finance, and management of water resources that underpin sustainability. Second, the Agency has issued a set of technical briefs that provide new guidance on important topics for developing and implementing WASH activities, as well as recommendations for activity design, implementation, and monitoring. Third, USAID has launched the Water Security, Sanitation and Hygiene Implementation Research Agenda that identifies and prioritizes sector-specific research questions to close lingering evidence gaps directly related to accomplishing USAID’s goal of increasing access to sustainable water and sanitation services. Finally, the Agency is rethinking its approach to sector targets and key performance indicators in its solicitations, recognizing that targets can cause perverse incentives to undermine sustainability from the outset. Doing so underscores the Agency’s commitment to sustainability and its willingness to be held accountable to deliver against those results.

For more information, take a look at the full evaluation or access other ex-post evaluations on Globalwaters.org.

By Abigail Jones and Elizabeth Jordan, Water and Sanitation Specialists, USAID Center for Water Security, Sanitation, and Hygiene

 

Reblog Ex-post Eval Week: Exiting For Sustainability by Jindra Cekan

Posted by on Jan 22, 2021 in climate change, ex-post evaluation, Resilience, SDGs, Sustainability | 0 comments

Ex-post Eval Week: Exiting For Sustainability by Jindra Cekan

Jindra Cekan
Jindra Cekan

Hello. My name is Jindra Cekan, and I am the Founder and Catalyst of Valuing Voices at Cekan Consulting LLC. Our research, evaluation and advocacy network have been working on post-project (ex-post) evaluations since 2013. I have loved giraffes for decades and fund conservation efforts (see pix).

Our planet is in trouble as are millions of species, including these twiga giraffes and billions of homo-sapiens. Yet in global development we evaluate projects based on their sectoral, e.g. economic, social, educational, human rights etc., results, with barely a glance at the natural systems on which they rest. IDEAS Prague featured Andy Rowe and Michael Quinn Patton who showed that I too have been blind to this aspect of sustainability.

I have argued ad nauseum that the OECD’s definition of projected sustainability and impact don’t give a hoot about sustaining lives and livelihoods.. If we did, we would not just claim we do ‘sustainable development’ and invest in ‘Sustainable Development Goals’ but go about proving how well, for how long, by whom, after closeout.

After hearing Rowe, I added to my Sustained Exit Checklists new elements about how we must evaluate Risks to Sustainability and Resilience to Shocks that included the natural environment. I added Adaptation to Implementation based on feedback on how much implementation would need to change based in part on climatic changes.

Consider these Valuing Voices Sustained Exit Elements table with information on commitments to sustainability and conditions for sustainability.
Source: Author’s Sustained Exit training (December 2020)

Lessons Learned:

Yet new evaluation thinking by Rowe, Michael Quinn Patton, Astrid Brouselle/ Jim McDavid take us a quantum leap beyond. We must ask how can any intervention be sustained without evaluating the context in which it operates. Is it resilient to environmental threats? Can participants adapt to shocks,? Have we assessed and mitigated the environmental impacts of our interventions? As Professor Brouselle writes, “changing our way of thinking about interventions when designing and evaluating them…. away from our many exploitation systems that lead to exhaustion of resources and extermination of many species.”

This 2020 new thinking includes ascertaining:

  1. (Andy Rowe) Ecosystems of biotic natural capital and abiotic natural capital (from trees to minerals) with effects on health, education, public safety/ climate risk and community development
  2. (Michael Quinn Patton) New Evaluative Criteria including: Transformation fidelity, adaptive sustainability, eco-efficiency full-cost accounting
  3. (Astrid Brouselle and Jim McDavid) Human systems that affect our interventions, including: Power relations, prosperity, equity and we need to make trade-offs between environment and development goals clear.

We have miles to go of systems and values to change. Please read this and let’s start sustaining NOW.

Rad Resources:

This week, AEA365 is celebrating Ex-post Eval Week during which blog authors share lessons from project exits and ex-post evaluations. Am grateful to the American Evaluation Association that we could share these resources…. 

Reblog Ex-post Eval Week: Measuring sustainability post-program –go in and stay for the learning! By Holta Trandafili

Posted by on Jan 22, 2021 in Accountability, ex-post evaluation, Learning, World Vision | 0 comments

Ex-post Eval Week: Measuring sustainability post-program –go in and stay for the learning! By Holta Trandafili

 Holta Trandafili
Holta Trandafili

Greetings, I am Holta Trandafili, a researcher and evaluator captivated by sustainability theories and the sustainment of results. I believe that a thoughtful, systematic inquiry of what happens after an intervention ends adds value to what we know about sustainability. Since 2015 I have co-led ten post-program evaluations (also known as ex-posts) in Uganda, Kenya, Sri Lanka, India, Myanmar, and Bolivia. Their findings point to questions and issues of theory, measurement, and sustainability expectations relevant to any program:

  • To what do we compare results to judge success? Is it that 60% of community groups or water points being operational three years after closeout a good result? Should it be 87% or 90%? Why? Should we use the end-line as the measuring yardstick, especially as contexts change? Whose view of success counts?
  • How long should we expect results, or community groups left behind, or activities to continue post-program? Two or ten years or Forever? Why?
  • Is going back once enough to make a judgment on sustainability? What would we find if we went back in 2020 where we evaluated ex-post in 2015 or even 2019?

Lessons Learned: Here are my reflections and resources on sustainability:

To the enthusiastic evaluators ready to start ex-posts

  • Lesson learned: Organizations often carry out ex-posts for accountability. However, greater wealth lays in learning. Make learning part of your evaluation objectives. It took my organization 5 years from the first ex-post to have more open conversations and share our sustainability learning on what to improve: how we design, transition, and measure programs’ impact. Now we are genuinely more accountable.
  • Get involved: Don’t lose heart if your first ex-posts prove difficult to conduct or have mixed results or unearth new questions and insights on sustainability. You are not alone. Find another evaluator that has gone through an ex-post experience and ask them to write a blog, present at a conference, write guidelines, attend a course, or merely meet to vent and dream.

To those already fighting to mainstream ex-post measurement in their organizations or their clients

Hot Tips:

  • Mainstreaming ex-post evaluations is commendable for any institution. In this process we should start making the case to pilot longitudinal ex-post measurements (i.e., going back not once but several points in time). We can truly unpack the issues of temporality and longevity for sustainment of results. See JICA’s example on ex-post monitoring.
  • Invest in theory-driven evaluations like Realist Evaluation to unpack the hidden mechanisms behind which different types of outcomes are sustained, asking: among whom, in what contexts, how, why?

Rad Resources:

ReBlog Ex-post Eval Week: Evaluating Peace for Sustainability by Peter Kimeu Ngui

Posted by on Jan 21, 2021 in Appreciative Inquiry, CARITAS, CRS, Justice, Kenya, peace building, Pope Francis, Sustainability | 0 comments

Reblog Ex-post Eval Week: Evaluating Peace for Sustainability by Peter Kimeu Ngui

Peter Kimeu Ngui
Peter Kimeu Ngui

Peace is with you. My name is Peter Kimeu Ngui, the Founder and CEO of Decent Living Institute of Organic Farming with extensive experience on reflective peace and partnership building in Africa.

Is there anything like a ‘just world’? Can peace be ‘sustainable’? These two closely related questions must be in mind of evaluators of peace for sustainability in a community long after the funding expires. According to Catholic Relief Services’ Justice Lens, three key conditions that must exist in the vision of a just world and used by all sustainable peace evaluators.:

(1) sustainable right relationship for all humanity and nature

(2) the transformation of injustices in society, structures and policies with

(3) fairness of rights and responsibilities for sustained peace to all creation

The process of reflective peace building evaluation, including participatory creation of sustainability indicators demands a collaborative effort by all the stakeholders. This includes victims, perpetrators, peace builders/ advocacy agents who are necessary to establish and maintain a just environment.

Step one of the process involves reflecting on eight keys principles drawn from Pope Francis’ letter that awaken the conscience and human rights standards which fit into an acronym (CORDSSSS) for easy remembrance; Common Good, Option for the Poor, Rights and Responsibilities, Dignity and Equality, Social nature, Subsidiarity, Stewardship and Solidarity. Each team reflects on:

  • the definition of each principle
  • discussing what their world would be like if the principle was fully lived
  • what would be sustainable peace indicators (e.g. Increased peaceful resolutions administered by the community; peaceful co-existence by people in violent conflicts; reduced rape)
  • what needs to be different in the world (location) they live in?

Step two of the process is scenario building using the Appreciative Inquiry method that engages the participants in the evaluation. This involves finding out what:

  • sustainable values, customs, and traditions
  • policies and structures/ institutions exist in the community that promote/ advocate, promote, establish and maintain, monitor, and evaluate for sustainable peace building
  • the participants are led to dreaming of a just environment in their community and what needs be improved and put in place for this scenario to be realized.

Lessons Learned:

The evaluation teams are always and immediately energized to pursue the best scenario for peacebuilding, using the resources locally available.  The reflective peacebuilding process engages participants on real issues that directly affects the community, that the community is fears, worries, concerned with and they are immediately empowered with knowledge, using familiar traditional approach to resolve the violent conflict, and they self-discover that power to transform is with those engaged in the conflict.

The possibility of a just world becomes a reality amassing greater numbers of individuals to work for peace and work through these processes.

Lessons learned are self-created from local case study evaluation processes and appeal greatly to most of the community members

Rad Resources:

ReBlog Ex-Post Eval Week Tips for Conducting an Ex-Post Evaluation by Wendi Bevins

Posted by on Jan 19, 2021 in ex-post evaluation, Lutheran World Relief, stakeholders, Tanzania, World Vision | 0 comments

Repost:

Reblog: Eval Week: Tips for Conducting an Ex-Post Evaluation by Wendi Bevins

Wendi Bevins
Wendi Bevins

I am Wendi Bevins, and when my boss told me in 2017 that we were going to conduct an ex-post evaluation, I was thrilled. At the time I was the Monitoring and Evaluation Manager at Lutheran World Relief[1] (LWR).

I was thrilled to go back to a closed project and evaluate it against our original theory of change, test our assumptions, and see if our approaches were sustainable beyond the end of a project. We could really learn some valuable lessons. And all those things were true, but they came alongside some useful lessons about human nature, which I offer here as suggestions to anyone who wants to conduct an ex-post.

Hot Tip: Manage Expectations

Ex-post results, like any evaluation’s results, will not be entirely clear, and some will be disappointing. In my own experience, I pinned my hopes on an assumption and was disappointed when the result was murky. These kinds of realities do not mean that an ex-post is a bad investment, but they do mean that stakeholders should be prepared early in the planning stage to manage their expectations. Then stakeholders are more likely to use the findings effectively where they do point to a clear (if unexpected) path forward and they aren’t tempted to dismiss all the findings out of hand. While that is true for nearly all evaluations, it is especially true for ex-posts because the stakes feel higher.

Lessons Learned: How you apply findings matters almost as much as the findings themselves

Your ex-post will definitely yield findings. You will not know ahead of time what those findings are, and as I learned, you should resist the urge to guess what they will be. However, having a way to apply the findings is important. The value of the ex-post is tied up in an organization’s ability to make use of its findings. In the case of our first ex-post evaluation, we applied what we learned to an organization-wide strategy that was being developed as the ex-post ended. Like other evaluation findings, ex-post findings can also be valuable in the development of new tools or the design of new programming, but even better because they give a longer time frame for reflection.

Rad Resources: Practical Tips and Examples

In partnership with World Vision US, LWR compiled a set of practical tips for planning and carrying out an ex-post evaluation. It reflects our lessons learned over 11 ex-post evaluations across the two organizations, such as selecting the right project; hiring an evaluator; and staying future-oriented. LWR also condensed the most transferrable lessons from our two ex-post evaluations into user-friendly summaries: the Tanzanian Grape Value Chain project and Nicaraguan Gender-Sensitive Food Security project.

I hope if you are fortunate enough to participate in an ex-post evaluation that these tips give you a good head start.


[1] Lutheran World Relief and IMA World Health have since created Corus International, an ensemble of for-profit and non-profit organizations with social impact missions.

Reblog:Ex-post Eval Week: Are we serious about project sustainability and exit? By Abu Ala Hasan

Posted by on Jan 18, 2021 in Aid effectiveness, Bangladesh, Exit strategies, Project cycle, Sustainable development | 0 comments

Ex-post Eval Week: Are we serious about project sustainability and exit? By Abu Ala Hasan

I am Abu Ala Hasan, an independent consultant. I am an anthropologist, have been working in the NGO sector for 17 years. A large portion of my work are evaluations and research.

In 2018 we (Jindra Cekan and I) were selected to design exit strategy and learning documentation for a client. When we started, we found, despite having a plan for devising exit strategy, the implementing organisations involved only paid attention to that at the end of the project, having only three months.

Most startling was the fact that the informants, ranging from partner NGO officials,  to community members involved gave us blank looks when asked about readiness for exit. Few NGO executives said that they would be able to continue some of the activities themselves. Almost none of the community members could answer exit questions; rather they tried to explain about the project services and their benefits.

People sitting around a table with large piece of paper with squares drawn in and round stickers in several of the squares.

Though it was thought provoking, it was not surprising at all due to existing NGO culture, where in practice, interventions are initiated and decisions made by the NGOs or funders, rather than the community.

So, why does it happen?  First and foremost, exit is not taken seriously. Rather than trying to ensure sustainability of the project from the beginning or designing the project to achieve it, it is added, as if, an auxiliary component; apparently to fulfil criteria by outsiders. The word ‘exit strategy’ was mentioned twice and ‘sustainability’ five times in the action plan, a 38- page document, and we saw few activities.

While it takes time, the trend of project cycle has been increasingly heading towards shorter duration. A decade ago most projects were planned for five years or longer but now one to three years is the norm. This is not only counterproductive for sustainability but also detrimental to bring about significant social change (outcome/impact). The community members’ reactions indicated that sustainability was unfamiliar; it appeared, they were not informed or there was no such discussion with them. Planning for sustainability was not taken seriously.

Hot Tips:

To ensure sustainability and proper exit:

  • There should be sincere commitment and effort towards sustainability and exit from NGOs
  • Sustainability and exit planning should be built in the project implementation process from design stage. Projects should be designed for sustainability and not ignored
  • Involvement of the community members, their voice and participation in decision making at all stages is very important for sustainability and proper exit
  • Except for emergency projects, development projects should be longer in duration and properly evaluated periodically

Rad Resources:

For exit and sustainability planning these resources could be helpful:

This week, AEA365 is celebrating Ex-post Eval Week during which blog 

Upcoming Jan Webinar: Lessons from Nordic / the Netherlands’ ex-post project evaluations: 14 Jan 2021

Posted by on Jan 9, 2021 in Denmark, Evaluation, ex-post evaluation, Finland, impact evaluation, Netherlands, Nordics, Norway, Sustainability, Sustainable development, Sweden, Valuing Voices | 0 comments

Upcoming Webinar: Lessons from Nordic / the

Netherlands’ ex-post project evaluations: 14 Jan 2021

In June-August 2020, Preston Stewart, our Valuing Voices intern, conducted through government databases of the four Nordic countries – Norway, Finland, Sweden, and Denmark – and the Netherlands to identify ex-post evaluations of government-sponsored projects. The findings and recommendations for action are detailed in a four-part white paper series, beginning with a paper called The Search.

We would like to invite you to a presentation of our findings, and a discussion of what can be learned about:

1) the search process,

2) how ex-post evaluations are defined and categorized,

3) what was done well by each country’s ex-posts,

4) sustainability-related findings and lessons, and

5) what M&E experts in each country can improve on ex-post evaluation practices.

One big finding is that there were only 32 evaluations that seemed to be ex-post project, and only 1/2 of them actually were at least 2 years after project closure.

 

We have many more lessons about conflicting definitions, that ex-post evaluation is not the norm in the evaluations processes of the five governments, that development programs could, if committed to ex-post evaluation, learn about sustainability by engaging with the findings from many more such evaluations, and to increase accountability to the public and for transparent learning, ex-post evaluations should be shared in public, easy-to-access online repositories.

Join us!

REGISTER: https://www.eventbrite.com/e/webinar-lessons-from-nordic-the-netherlands-ex-post-project-evaluations-tickets-132856426147 

Your ticket purchase entitles you to the webinar, its meeting recording, associated documents, and online Sustainability Network membership for resources and discussion. Payment on a sliding, pay-as-you-can scale.

Interview repost: Why Measure & Evaluate Corporate Sustainability Projects?

Posted by on Dec 18, 2020 in CSR, Evaluation, Sustainability, Sustainable development | 0 comments

Interview repost: Why Measure & Evaluate Corporate Sustainability Projects?

A CSR firm in Central Europe asked me to talk about sustainability, evaluation, and Corporate Social Responsibility. We had a terrific interview – please click thru to: https://www.besmarthead.com/cs/media/post/smartheadtalks-2-jindra-cekan-eng

 

 

The topics we discussed include:

* Sustainability in CSR involves value creation through benefits-creation for both companies and consumers

* Projects feeding into the United Nations Sustainable Development Goals may not be sustained over the long-term and we must return to evaluate what could be sustained and what emerged ex-post

* Scale measurable and meaningful impact investment through the transformation of development nonprofits’ programming + approach

* Three activities in sustainability that companies can start doing right away, that you would recommend to any company anywhere in the world

And Very Happy Holidays to all… may our lives and world be more sustainable in 2021….

 

 

Reblog: ITAD/CRS “Lessons from an ex-post evaluation – and why we should do more of them”

Posted by on Dec 10, 2020 in Catholic Relief Services (CRS), Credit, Evaluation, ex-post evaluation, microcredit, Microenterprise, Sustainable development | 0 comments

Reblog: ITAD/CRS “Lessons from an ex-post evaluation – and why we should do more of them”

Reposted from: https://www.itad.com/article/lessons-from-an-ex-post-evaluation-and-why-we-should-do-more-of-them/

Even as evaluation specialists, rarely do we get the chance to carry out ex-post evaluations. We recently carried out an ex-post evaluation of Catholic Relief Services’ (CRS) Expanding Financial Inclusion (EFI) programme and believe we’ve found some key lessons that make the case for more ex-post evaluations.

We’ll be sharing learning from the evaluation alongside CRS colleagues at the Savings Led Working Group session on Members Day of the SEEP Annual Conference – so pop by if you would like to learn more.

What is an ex-post evaluation?

Ex-post evaluations are (by definition) done after the project has closed. There is no hard and fast rule on exactly when an ex-post evaluation should be done but as the aim of an ex-post evaluations is to assess the sustainability of results and impacts, usually some time will need to have passed to make this assessment.

A little bit about EFI

EFI was a Mastercard Foundation-funded program in Burkina Faso, Senegal, Zambia and Uganda whose core goal was to ensure that vulnerable households experienced greater financial inclusion. Within EFI, Private Service Providers (PSPs) formed and facilitated savings groups using CRS’ Savings and Internal Lending Communities (SILC) methodology, with the SILC groups responsible for paying the PSP a small fee for the services that they provide.

This payment is intended to improve sustainability by incentivising the groups’ facilitators to form and train new groups, as well as providing continued support to existing groups, beyond the end of the project.

A little bit about the evaluation

So, if the aim of the PSP model is sustainability, you need an evaluation that can test this! Evaluation at the end of project implementation can assess indications of results that might be sustained into the future. However, if you wait until some time has passed after activities have ended, then there is much clearer evidence on which activities and results are ongoing – and how likely these are to continue. Uganda was also a great test case for the evaluation because CRS hadn’t provided any follow-on support.

Our evaluation set out to assess the extent to which the EFI-trained PSPs and their SILC groups were still functioning 19 months after the programme ended and the extent to which the PSP model had contributed to the sustainability of activities and results.

What the ex-post evaluation found

We found a handful of findings that were only possible because it was an ex-post evaluation:

  • There were 56% more reported groups among the sampled PSPs at the time of data collection than there were at the end of the project.
  • Half of the PSP networks established within the sample are still functioning (to some extent).
  • PSPs continued to receive remuneration for the work that they did, 19 months after project closure. However, there were inconsistencies in frequency and scale of remuneration, as well as variation in strategies to sensitize communities on the need to pay.

This only covers a fraction of the findings but we were able to conclude that the PSP model appeared to be highly sustainable. The evaluation also found that there were challenges to sustainability which could be addressed in future delivery of the PSP model. Significantly, the PSP model was designed with sustainability in mind – and this evaluation provides good evidence that PSPs were still operating 19 months after the end of the project.

What made the evaluation possible

We get it. It isn’t always easy to do ex-post evaluations. Evaluations are usually included in donor-implementer contracts, which end shortly after the project ends, leaving implementers without the resources to go back and evaluate 18 months later. This often results in a lack of funding and an absence of project staff. This is also combined with new projects starting up, obscuring opportunities for project-specific findings and learning as it’s not possible to attribute results to a specific project.

In many ways, we were lucky. Itad implements the Mastercard Foundations Savings Learning Lab, a six-year initiative that supports learning among the Foundation’s savings sector portfolio programmes – including EFI. EFI closed in the Learning Lab’s second year and with support from the Foundation and enthusiasm from CRS, we set aside some resource to continue this learning post-project. So, we had funding!

We also worked with incredibly motivated ex-EFI, CRS staff who made time to actively engage in the evaluation process and facilitate links to the PSP network, PSPs and SILC group members. So, we had the people!

And, no-one had implemented a similar PSP model in supported districts of Uganda since the end of EFI. So, we were also able to attribute!

Why we should strive to do more ex-post evaluations

Despite these challenges, and recognising it isn’t always easy, doesn’t mean it is not possible. And with projects like EFI where sustainability was central to its model, we would say it’s essential to assess whether the programme worked and how the model can be improved.

Unfortunately, practitioners and evaluators can shout all we like but the onus is on funders. We need funders to carve out dedicated resource for ex-post evaluations. This is even more important for programmes that have the development of replicable and sustainable models at their core. For some projects, this can be anticipated – and planned for – at project design stage. Other projects may show promise for learning on sustainability, unexpectedly, during implementation. Dedicated funding pots or call-down contracts for ex-post evaluations are just a couple of ways donors might be able to resource ex-post evaluations when there is a clear need for additional learning on the sustainability project results.

This learning should lead to better decision making, more effective use of donor funds and ultimately, more sustainable outcomes for beneficiaries.”

Other Findings:

Some of the other findings of this report on Financial Inclusion are:

RESOURCES: “Finding 1.iii. PSPs continue to receive remuneration for the work that they do; however, there are inconsistencies in frequency and scale of remuneration, as well as variety in strategies to sensitize communities on the need to pay.”

CAPACITIES: “Finding 3.ii. All networks included a core function of “collaboration, information-sharing and problemsolving”; however, networks were not sufficiently supported or incentivized to fulfill complex functions, such as PSP quality assurance or consumer protection, and their coverage area and late implementation limited the continued functioning of networks.”

PARTNERSHIP: “Finding 2.i. Only four of the 24 groups are clearly linked with other stakeholders and two were supported by EFI to create these linkages.”

Consider doing one!