Ex-post Eval Week: Exiting For Sustainability by Jindra Cekan
Reblogged from AEA: https://aea365.org/blog/ex-post-eval-week-exiting-for-sustainability-by-jindra-cekan-2/ January 22, 2021
Hello. My name is Jindra Cekan, and I am the Founder and Catalyst of Valuing Voices at Cekan Consulting LLC. Our research, evaluation and advocacy network have been working on post-project (ex-post) evaluations since 2013. I have loved giraffes for decades and fund conservation efforts (see pix).
Our planet is in trouble as are millions of species, including these twiga giraffes and billions of homo-sapiens. Yet in global development we evaluate projects based on their sectoral, e.g. economic, social, educational, human rights etc., results, with barely a glance at the natural systems on which they rest. IDEAS Prague featured Andy Rowe and Michael Quinn Patton who showed that I too have been blind to this aspect of sustainability.
I have argued ad nauseum that the OECD’s definition of projected sustainability and impact don’t give a hoot about sustaining lives and livelihoods.. If we did, we would not just claim we do ‘sustainable development’ and invest in ‘Sustainable Development Goals’ but go about proving how well, for how long, by whom, after closeout.
After hearing Rowe, I added to my Sustained Exit Checklists new elements about how we must evaluate Risks to Sustainability and Resilience to Shocks that included the natural environment. I added Adaptation to Implementation based on feedback on how much implementation would need to change based in part on climatic changes.
Yet new evaluation thinking by Rowe, Michael Quinn Patton, Astrid Brouselle/ Jim McDavid take us a quantum leap beyond. We must ask how can any intervention be sustained without evaluating the context in which it operates. Is it resilient to environmental threats? Can participants adapt to shocks,? Have we assessed and mitigated the environmental impacts of our interventions? As Professor Brouselle writes, “changing our way of thinking about interventions when designing and evaluating them…. away from our many exploitation systems that lead to exhaustion of resources and extermination of many species.”
This 2020 new thinking includes ascertaining:
(Andy Rowe) Ecosystems of biotic natural capital and abiotic natural capital (from trees to minerals) with effects on health, education, public safety/ climate risk and community development
(Astrid Brouselle and Jim McDavid) Human systems that affect our interventions, including: Power relations, prosperity, equity and we need to make trade-offs between environment and development goals clear.
We have miles to go of systems and values to change. Please read this and let’s start sustaining NOW.
Rowe, A. (2019). Sustainability-read evaluation: A call to action. In G. Jules (Ed.) Evaluating Sustainability: Evaluative Support for Managing Processes in the Public Interest. New Directions for Evaluation, 162, 29-48.
This week, AEA365 is celebrating Ex-post Eval Week during which blog authors share lessons from project exits and ex-post evaluations. Am grateful to the American Evaluation Association that we could share these resources….
Ex-post Eval Week: Measuring sustainability post-program –go in and stay for the learning! By Holta Trandafili
Reblog from AEA: https://aea365.org/blog/ex-post-eval-week-measuring-sustainability-post-program-go-in-and-stay-for-the-learning-by-holta-trandafili/ January 21, 2021
Greetings, I am Holta Trandafili, a researcher and evaluator captivated by sustainability theories and the sustainment of results. I believe that a thoughtful, systematic inquiry of what happens after an intervention ends adds value to what we know about sustainability. Since 2015 I have co-led ten post-program evaluations (also known as ex-posts) in Uganda, Kenya, Sri Lanka, India, Myanmar, and Bolivia. Their findings point to questions and issues of theory, measurement, and sustainability expectations relevant to any program:
To what do we compare results to judge success? Is it that 60% of community groups or water points being operational three years after closeout a good result? Should it be 87% or 90%? Why? Should we use the end-line as the measuring yardstick, especially as contexts change? Whose view of success counts?
How long should we expect results, or community groups left behind, or activities to continue post-program? Two or ten years or Forever? Why?
Is going back once enough to make a judgment on sustainability? What would we find if we went back in 2020 where we evaluated ex-post in 2015 or even 2019?
Lessons Learned:Here are my reflections and resources on sustainability:
To the enthusiastic evaluators ready to start ex-posts
Lesson learned: Organizations often carry out ex-posts for accountability. However, greater wealth lays in learning. Make learning part of your evaluation objectives. It took my organization 5 years from the first ex-post to have more open conversations and share our sustainability learning on what to improve: how we design, transition, and measure programs’ impact. Now we are genuinely more accountable.
Get involved: Don’t lose heart if your first ex-posts prove difficult to conduct or have mixed results or unearth new questions and insights on sustainability. You are not alone. Find another evaluator that has gone through an ex-post experience and ask them to write a blog, present at a conference, write guidelines, attend a course, or merely meet to vent and dream.
To those already fighting to mainstream ex-post measurement in their organizations or their clients
Mainstreaming ex-post evaluations is commendable for any institution. In this process we should start making the case to pilot longitudinal ex-post measurements (i.e., going back not once but several points in time). We can truly unpack the issues of temporality and longevity for sustainment of results. See JICA’s example on ex-post monitoring.
Invest in theory-driven evaluations like Realist Evaluation to unpack the hidden mechanisms behind which different types of outcomes are sustained, asking: among whom, in what contexts, how, why?
Reblog Ex-post Eval Week: Evaluating Peace for Sustainability by Peter Kimeu Ngui
Reblogged from https://aea365.org/blog/ex-post-eval-week-evaluating-peace-for-sustainability-by-peter-kimeu-ngui/ January 20, 2021
Peace is with you. My name is Peter Kimeu Ngui, the Founder and CEO of Decent Living Institute of Organic Farming with extensive experience on reflective peace and partnership building in Africa.
Is there anything like a ‘just world’? Can peace be ‘sustainable’? These two closely related questions must be in mind of evaluators of peace for sustainability in a community long after the funding expires. According to Catholic Relief Services’ Justice Lens, three key conditions that must exist in the vision of a just world and used by all sustainable peace evaluators.:
(1) sustainable right relationship for all humanity and nature
(2) the transformation of injustices in society, structures and policies with
(3) fairness of rights and responsibilities for sustained peace to all creation
The process of reflective peace building evaluation, including participatory creation of sustainability indicators demands a collaborative effort by all the stakeholders. This includes victims, perpetrators, peace builders/ advocacy agents who are necessary to establish and maintain a just environment.
Step one of the process involves reflecting on eight keys principles drawn from Pope Francis’ letter that awaken the conscience and human rights standards which fit into an acronym (CORDSSSS) for easy remembrance; Common Good, Option for the Poor, Rights and Responsibilities, Dignity and Equality, Social nature, Subsidiarity, Stewardship and Solidarity. Each team reflects on:
the definition of each principle
discussing what their world would be like if the principle was fully lived
what would be sustainable peace indicators (e.g. Increased peaceful resolutions administered by the community; peaceful co-existence by people in violent conflicts; reduced rape)
what needs to be different in the world (location) they live in?
Step two of the process is scenario building using the Appreciative Inquiry method that engages the participants in the evaluation. This involves finding out what:
sustainable values, customs, and traditions
policies and structures/ institutions exist in the community that promote/ advocate, promote, establish and maintain, monitor, and evaluate for sustainable peace building
the participants are led to dreaming of a just environment in their community and what needs be improved and put in place for this scenario to be realized.
The evaluation teams are always and immediately energized to pursue the best scenario for peacebuilding, using the resources locally available. The reflective peacebuilding process engages participants on real issues that directly affects the community, that the community is fears, worries, concerned with and they are immediately empowered with knowledge, using familiar traditional approach to resolve the violent conflict, and they self-discover that power to transform is with those engaged in the conflict.
The possibility of a just world becomes a reality amassing greater numbers of individuals to work for peace and work through these processes.
Lessons learned are self-created from local case study evaluation processes and appeal greatly to most of the community members
Ex-post Eval Week: Are we serious about project sustainability and exit? By Abu Ala Hasan
Published on the American Evaluation Association AEA365blog series: https://aea365.org/blog/ex-post-eval-week-are-we-serious-about-project-sustainability-and-exit-by-abu-ala-hasan/ January 18, 2021
I am Abu Ala Hasan, an independent consultant. I am an anthropologist, have been working in the NGO sector for 17 years. A large portion of my work are evaluations and research.
In 2018 we (Jindra Cekan and I) were selected to design exit strategy and learning documentation for a client. When we started, we found, despite having a plan for devising exit strategy, the implementing organisations involved only paid attention to that at the end of the project, having only three months.
Most startling was the fact that the informants, ranging from partner NGO officials, to community members involved gave us blank looks when asked about readiness for exit. Few NGO executives said that they would be able to continue some of the activities themselves. Almost none of the community members could answer exit questions; rather they tried to explain about the project services and their benefits.
Though it was thought provoking, it was not surprising at all due to existing NGO culture, where in practice, interventions are initiated and decisions made by the NGOs or funders, rather than the community.
So, why does it happen? First and foremost, exit is not taken seriously. Rather than trying to ensure sustainability of the project from the beginning or designing the project to achieve it, it is added, as if, an auxiliary component; apparently to fulfil criteria by outsiders. The word ‘exit strategy’ was mentioned twice and ‘sustainability’ five times in the action plan, a 38- page document, and we saw few activities.
While it takes time, the trend of project cycle has been increasingly heading towards shorter duration. A decade ago most projects were planned for five years or longer but now one to three years is the norm. This is not only counterproductive for sustainability but also detrimental to bring about significant social change (outcome/impact). The community members’ reactions indicated that sustainability was unfamiliar; it appeared, they were not informed or there was no such discussion with them. Planning for sustainability was not taken seriously.
To ensure sustainability and proper exit:
There should be sincere commitment and effort towards sustainability and exit from NGOs
Sustainability and exit planning should be built in the project implementation process from design stage. Projects should be designed for sustainability and not ignored
Involvement of the community members, their voice and participation in decision making at all stages is very important for sustainability and proper exit
Except for emergency projects, development projects should be longer in duration and properly evaluated periodically
For exit and sustainability planning these resources could be helpful:
“NGO Research Culture and its Implication in Bangladesh: An Insider’s Perspective”, The Oriental Anthropologist, Vol. 14, No. 1, 2014, Pages 1-12, OICSR, Allahabad, India
Netherlands’ ex-post project evaluations: 14 Jan 2021
In June-August 2020, Preston Stewart, our Valuing Voices intern, conducted through government databases of the four Nordic countries – Norway, Finland, Sweden, and Denmark – and the Netherlands to identify ex-post evaluations of government-sponsored projects. The findings and recommendations for action are detailed in a four-part white paper series, beginning with a paper called The Search.
We would like to invite you to a presentation of our findings, and a discussion of what can be learned about:
1) the search process,
2) how ex-post evaluations are defined and categorized,
3) what was done well by each country’s ex-posts,
4) sustainability-related findings and lessons, and
5) what M&E experts in each country can improve on ex-post evaluation practices.
One big finding is that there were only 32 evaluations that seemed to be ex-post project, and only 1/2 of them actually were at least 2 years after project closure.
We have many more lessons about conflicting definitions, that ex-post evaluation is not the norm in the evaluations processes of the five governments, that development programs could, if committed to ex-post evaluation, learn about sustainability by engaging with the findings from many more such evaluations, and to increase accountability to the public and for transparent learning, ex-post evaluations should be shared in public, easy-to-access online repositories.
Your ticket purchase entitles you to the webinar, its meeting recording, associated documents, and online Sustainability Network membership for resources and discussion. Payment on a sliding, pay-as-you-can scale.
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).