The document(s) cover:
1) the search process,
2) how ex-post evaluations are defined and categorized,
3) what was done well by each country’s ex-posts,
4) sustainability-related findings and lessons, and
5) what M&E experts in each country can improve on ex-post evaluation practices.
6) Case studies of the Netherlands, Norway, and Finland
One big finding is that there were only 32 evaluations that seemed to be ex-post project, and only 16 of them actually were at least 2 years after project closure.
Executive Summary/ Abstract of the set of White Papers
Ex-post completion or ex-post project evaluations are surprisingly rare given widespread commitments to ‘sustainable development’. Given the well-known commitment of the Nordic countries of Norway, Finland, Sweden, and Denmark as well as the Netherlands, Valuing Voices, a consulting firm specializing in ex-post project evaluation, chose these countries as case studies. We hired an intern from Harvard College, who searched the national databases (in English only) and presented these results. The findings were surprising. There were far fewer ex-posts in total than expected. Those that were included a surprising number that was co-evaluated alongside final evaluations, although final evaluation illuminates relevance, effectiveness, efficiency (and the new OECD criteria of coherence), while ex-post project evaluation illuminates sustainability and impact two or more years after closure. There were lessons for Ministries about the searchability as well as the quality (and in the case of Denmark and in part, Sweden), the dearth of ex-posts at all. Overall, the recommendations for the five countries researched for this paper include better and more standardized definitions of ex-post project, greater absolute numbers done, transparent sharing of those done in publicly searchable manners, methodologically clear comparisons (baseline and midterm), and clarity in differentiating different evaluations.
There are caveats to these papers. This research was privately by Valuing Voices so the samples featured from the five countries were limited to public sources about ex-post and ex-post evaluations. While we reached out to Ministry evaluative staff in all countries, only two made themselves available for consultation (the Netherlands and Finland) and provided a handful of additional ex-post evaluations. This paper focuses on what such research yielded, not definitive findings of programs or multi-year country strategies that are funded for 20-30 years continuously, nor projects funded by country-level embassies which did not feature on the Ministry site. We focus on project bilateral project evaluations, not multilateral funding of sectors. We also presented these findings in a 2021 webinar, during which we received input that Sweden’s EBA has a (non-project) portfolio of ‘country evaluations’ which looked back over 10 or even 20-year time horizons.. Still, we found very few ex-post evaluations at EBA (see cite in Search, below). No input was received by Norway or Denmark.
There are four papers in this series combined in this White Paper. First, the search across databases of all five Nordics (Norway, Finland, Sweden, and Denmark) and the Netherlands (pages 1-21). This is followed by country-specific, detailed, papers for those with ex-posts: The Netherlands (pages 22-37), Norway (pages 38-47), and Finland (pages 48-59). We look forward to your feedback. Jindra@ValuingVoices.com for the Valuing Voices team.
Let us know what you think… Thanks, Jindra & Preston Stewart
Ex-post Eval Week: Are we serious about project sustainability and exit? By Abu Ala Hasan
Published on the American Evaluation Association AEA365blog series: https://aea365.org/blog/ex-post-eval-week-are-we-serious-about-project-sustainability-and-exit-by-abu-ala-hasan/ January 18, 2021
I am Abu Ala Hasan, an independent consultant. I am an anthropologist, have been working in the NGO sector for 17 years. A large portion of my work are evaluations and research.
In 2018 we (Jindra Cekan and I) were selected to design exit strategy and learning documentation for a client. When we started, we found, despite having a plan for devising exit strategy, the implementing organisations involved only paid attention to that at the end of the project, having only three months.
Most startling was the fact that the informants, ranging from partner NGO officials, to community members involved gave us blank looks when asked about readiness for exit. Few NGO executives said that they would be able to continue some of the activities themselves. Almost none of the community members could answer exit questions; rather they tried to explain about the project services and their benefits.
Though it was thought provoking, it was not surprising at all due to existing NGO culture, where in practice, interventions are initiated and decisions made by the NGOs or funders, rather than the community.
So, why does it happen? First and foremost, exit is not taken seriously. Rather than trying to ensure sustainability of the project from the beginning or designing the project to achieve it, it is added, as if, an auxiliary component; apparently to fulfil criteria by outsiders. The word ‘exit strategy’ was mentioned twice and ‘sustainability’ five times in the action plan, a 38- page document, and we saw few activities.
While it takes time, the trend of project cycle has been increasingly heading towards shorter duration. A decade ago most projects were planned for five years or longer but now one to three years is the norm. This is not only counterproductive for sustainability but also detrimental to bring about significant social change (outcome/impact). The community members’ reactions indicated that sustainability was unfamiliar; it appeared, they were not informed or there was no such discussion with them. Planning for sustainability was not taken seriously.
Hot Tips:
To ensure sustainability and proper exit:
There should be sincere commitment and effort towards sustainability and exit from NGOs
Sustainability and exit planning should be built in the project implementation process from design stage. Projects should be designed for sustainability and not ignored
Involvement of the community members, their voice and participation in decision making at all stages is very important for sustainability and proper exit
Except for emergency projects, development projects should be longer in duration and properly evaluated periodically
Rad Resources:
For exit and sustainability planning these resources could be helpful:
“NGO Research Culture and its Implication in Bangladesh: An Insider’s Perspective”, The Oriental Anthropologist, Vol. 14, No. 1, 2014, Pages 1-12, OICSR, Allahabad, India
Building the Evidence Base for Post Project Evaluation:
A report to the Faster Forward Fund
We are delighted to share Valuing Voices’ report on the value added of post-project evaluation, which compares findings from eight end-of-project and subsequent post project evaluations [1]. Many of you are aware of how rarely post project evaluations are undertaken. As a result, there is little real evidence about project impact on long-term sustainability. Valuing Voices received a grant from Michael Scriven’s Faster Forward Fund to begin to address this gap.
Our findings show that post project evaluations can contribute to better understanding of sustainability impacts, and reveal unexpected and emerging outcomes years after project close. They also indicate ways in which we can design and implement for sustainability.
Finding suitable projects for this review was difficult because so few post project evaluations are done, fewer are publically available, and fewer still had comparable final evaluations and included local voices. Agencies that fund post project evaluations offer a range of reasons for doing so: to learn, to promote a success, to inform replication or scale, to provide justification for future funding, to promote accountabilities. However, many funding agencies consider post project evaluation a luxury or not necessary. JICA and OECD are notable exceptions in this regard.
Highlights include:
The review highlights the range of methods that have been used in post project evaluations, and point to the advantages of planning for sustainability measurement from the outset of the project.
The cases reviewed in the study highlight the (sometime dramatic) difference between the anticipated trajectory of a project, what is happening as the project ends, and what actually continued, was adapted, ceased or changed course after close out.
Taxonomies, knowledge management about evaluation, data retrieval/ retention, analysis, use and dissemination are elements of sustained impact evaluation that require attention.
Little documentation is available about how post project evaluations have actually informed and influenced organizational learning, sectoral dialogue or future programming.
Post project evaluations shed particularly interesting light on what emerged post-project that was entirely due to the efforts and resources of participants and partners after project investments stopped. More on these Sustained and Emerging Impacts Evaluations (SEIEs) at Better Evaluation.
We welcome your comments on this report and checklists, and encourage you to share it in your networks and get us feedback on their use. Please use the report and findings to advocate for more post project sustainability impact evaluations which will contribute to greater evidence-based learning about project sustainability. Valuing Voices is among a handful of organizations who do post-project evaluations and we can either conduct one or refer you to another who does.
Thank you,
Laurie Zivetz, MPH, PhD and Jindra Cekan, PhD, with Kate Robins, MPH, PhD of Valuing Voices
Presenting Lessons on (post-project) Sustained and Emerging Impact Evaluations from the U.S. AEA Conference
Dear readers, attached please find the Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation presentation we did last week at the American Evaluation Association (AEA) conference in Atlanta GA [1]. I had the pleasure of co-presenting with Beatrice Lorge Rogers PhD, Professor, Friedman Nutrition School, Tufts University (aka the famous Food for Peace/ Tufts Exit Strategy study [2]), Patricia Rogers PhD, Director, BetterEvaluation, Professor, Australia and New Zealand School of Government (where we recently published guidance on SEIE [3]) and Laurie Zivetz PhD, International Development Consultant and Valuing Voices evaluator.
We integrated our presentations from Africa, Asia and Latin America into this fascinating overview:
1.Sustained and Emerging Impact Evaluation: global context
2.SEIE: definitions and methods
3.Case studies: findings from post-project evaluations
4.Designing an SEIE: Considerations
5.Q&A — which fostered super comments, but since you couldn’t come, please tell us what you think and what questions you have…
There are amazing lessons to learn about design, implementation, M&E from doing post-project evaluation. We have also grown in appreciating that sustainability can be tracked throughout the project cycle, not just during post-project SEIE evaluation.
We’ll be building this into a white paper or a … (toolkit? webinar series? training? something else?). What’s your vote ___? (I know in this US election season, so… :)).
What would you like to get to support your learning about Sustained and Emerging Impact Evaluations? Look forward to hearing from you- Jindra@ValuingVoices.com
[2] Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
Sustaining projects during and after Implementation: Does gender count?
Rutere Kagendo from Ronto Research and Valuing Voices
Gender as a concept is one of the many kinds of indicators of change measured in development projects. However, like many others, its reporting in many cases is limited to the number of women and men participating in the project. Whereas participation by gender is an important variable in measuring community engagement and inclusiveness in a project, this measure is more focused at the registration level and the mere fact that women’s or men’s names appear in the registers. However, numbers only are not a clear indicator of participation and the extent to which it happens.
Gender reporting, as shown above, can be too simplified. There is also ongoing misunderstanding surrounding gender, which only separates genders and omits a gendered analysis that would help understand the relationships between men and women, boys and girls within the dynamics of a project. This can include differentiated decisions about how resources are allocated within household expenditures (e.g. women overseeing food consumption, health and children’s expenses versus men overseeing large investments into livelihoods) and division of labor (e.g. men ploughing and planting fields and women weeding and conserving crops leading to a family harvest). Gender negotiations are richer than numbers of men and women participating in projects.
As we consider the various stages of a project, starting from design, implementation to evaluation and sustainability, it is critical to assess the role of gender in each of these stages. It is obvious that women and men play important roles in projects but what is not clear is how these roles build, complement and sustain the project. The questions that come into mind after interacting with communities during project evaluations (mid, endline, impact and sustainability) are:
How do the various roles played by both men and women, and their interaction with each other shape the outcome of a project?
Does gender contribute to project sustainability?
These are important questions that need to be answered through more research and writing but there are early insights I have learned:
1. Resilience in roles during the project life cycle
Studies highlighting the roles played by both women and men in projects shed light on how gender shapes project outcomes. I once attended a dairy project launch exercise in Kenya, my home country, and was surprised by what I saw. All the men sat at the front and the women took a back seat with their young children. The women remained silent the entire time the meeting was held, and efforts to involve them were unsuccessful. I later learned that in that community, women do not share their views in the presence of men. However, I left with one question in my mind, how will this project succeed if only one party is going to be talking and making decisions? Was there no mechanism for women to share their views as well? Is it not going to become a project that benefits men only? Years later, I got an opportunity to evaluate the project. I held both men and women focused group discussions separately. I found that firstly, men may be the gatekeepers but may not necessarily have the resilience needed to push through the whole project cycle. For example in this particular project, men did not continuously stay with the project, as noted by one female group participant:
“When this project started, we were both men and women. However, men decided to pull out because they thought milk was a women’s commodity and there was no money [to be made] in it. The women got organized in groups and started selling the milk in bulk. We made money and within no time, the men came back and took over the milk business. If we women had not stayed in the project, the milk business would not have been realized although we lost it to the men”. In another dairy-related project elsewhere, the men were the cattle and milk owners so they sold all the milk. But during a season when the market was flooded with milk (milk glut), the men abandoned the milk business. The women took it up and tried to sell the milk whenever possible until the milk glut’ season passed, only for the men to get back to the milk business again.
Whether it is a good idea or not for men to have taken the milk business away from women in both of these cases is subject for discussion, but the main issue here is the role played by the women to sustain the project long enough to grow it from an idea and push it into a viable business that attracted the men back into the project. Indeed one wonders if the project would have grown if the women had not remained in the project. Valuing Voices at Cekan Consulting found the same in an all-women and highly successful sesame seed production Catholic Relief Services project in the Gambia. Women had grown the communal agribusiness from microenterprise size to being so successful that they became of interest to local banks. At that point, men in the communities began to take the project, and savings, over. Valuing Voices has documented savings and loans sustained post-project, and even scaled up by women in several countries.
Also thinking about the initial community entry meetings and seeing how unengaged and distant the women appeared, one would have dismissed their role in the project. As it turned out, women became the engine of this project. Surface impressions may not hold true.
2. Saving the project for whose benefit?
In another project, community members were encouraged and supported to start savings and lending groups to run their small businesses. The qualitative discussions in the final evaluation revealed that at the end of the project, it was perceived as more of a women’s project in spite of being targeted to men. This is because almost all the men had left the project after taking loans and failing to repay. The women, especially those whose spouses had taken the loans took over the burden of repaying the loans which saved the project.
Borrowing from these examples one is tempted to ask, what happens in those projects that target men only or where women get demoralized and leave? Do they fail at a higher rate altogether than those that include women? Arguing along these lines one would ask, what are the driving forces for men and women to remain or quit from project activities and how has this been addressed so far in project designs?
In most of the evaluation projects Ronto Research has conducted around Africa, men may take a lead in project implementation depending on their expected quick/ shorter-term gains/ expectations which determine whether they remain active or not. On the other hand, women seem to get into projects with a determined resolve to see their problems solved, no matter how long it takes. As a result, they hold onto the project activities despite the challenges that come along. It is observable though, that as they progress and begin to take hold and show signs of success, the men get attracted back.
In most of the project evaluations I have participated in, there are always few men who are loyal and remain part and parcel of the project. Pivotally, we also need to understand the role of these few men who have remained in these projects with the women the whole time. This can be either because they hold positions or just because they believed in the projects and they support the women with ideas or just their mere presence in the project activities. These men seem to play a critical role in encouraging the women to push on. Their role cannot be ignored and there is need to understand them better in terms of their opinions. Why do they remain with the women even when there seems to be few benefits forthcoming to them, whereas their fellow men pull out? In an interview with women in Tanzania on an agricultural project, some women felt that the few men who had been left in their group were very instrumental in their achievements by giving the project an identity in as far as it being a community and not a ‘women-only’ project was concerned.
3.Gender- Informing evaluation
Thirdly, while conducting evaluation interviews, gender counts. Typically, women and men are asked about results in gender-specific groups. At the household level, the preferred interviewee is the household head. In most cases, they would be the direct ‘beneficiaries’ (participants) of the project information on behalf of their households. Due to the cultural set-ups in much of Africa, in married families, men are automatically the household head, which would mean they would be the target interviewees. However, our experience during interviews either through Focus Group Discussions or on a one-on-one situation is that consulting with women as well as considering differentiated gender roles and gender relations are key missing ingredients. For often during interviews, the men end up involving their spouses for details about projects, in part because men might not consistently follow up the project activities and therefore might not have all the details.
There is also a possibility that although it is the men who are registered as participants, in actual sense it is the women who participate and therefore have the detailed information about some issues being discussed, so it is important to check who is the actual ‘information bank’ (and day-to-day participant) of the household for the project. This issue needs to be explored in-depth both in regular project evaluation and more needs to be learned about how this can be used to inform sustainability studies. Asking only men or only women may limit evaluative learning.
4. Sustaining results – which gender is best?
What can we learn from the projects that have been completed so far years after they close out? When projects come to an end, communities are left to experience and foster the project impacts on their own. They learn a lot on their own post-project and this could be an important source of information on sustainability, but what mechanisms can be put in place for these ‘information banks’ to share the information long after projects end? They may be the only key informants that remain (as partners may have long gone onto other projects far away); before the knowledge they hold that can be eroded with time. It would be interesting to compare the information held by men and women about outcomes and impacts being sustained, stopped or new ones emerging. The question on information especially related to project sustainability is going to be very critical considering that most of this information has to be sought from community members post-project period.
Does gender play a role in holding a project together/implementation?
What role does gender play in project sustainability?
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).
