Longing to do an Ex-post Sustainability Evaluation? How to support this work…

by | May 18, 2015 | Dissemination, Feedback loops, innovation, local capacity building, Local Participants, post-project evaluation, Sustainability, Sustainable development

Longing to do an Ex-post Sustainability Evaluation? How to support this work…

Just back from Niger where Catholic Relief Services, Rutere Kagendo and I are doing a fascinating post-project evaluation.  Fresh on my mind is the commitment we all had to make to get this quite ground-breaking research going.  Here is the full report, but there are three kinds of conditions we found were integral for success: client-ValuingVoices match; project and site selection; and resources.

Client – Valuing Voices match:

  1. The study needs to be appreciated as innovative, adding to the program quality and learning of the organization so funding is provided and there is in-house interest in the findings;
  2. The local office needs to allocate staff and technical time to support the study technically and logistically (see below);
  3. Shared clarity is needed among all involved that such a study looks for self-sustained activities and outcomes. While there are lessons that emerge about the quality of implementation, its focus is what participants and their country-partners could continue themselves after project close-out and withdrawal of resources. It also can include lessons about what the local non-profit and the national stakeholders are doing to support community success (or not) and unexpected outcomes. Our clients need an openness to honestly seeing what was not sustained and exploring why;
  4. While Valuing Voices provides expertise given review of the handful of post-project and exit evaluations that exist, client is interested in sharing findings and advocating to donors to fund more of these studies;
  5. Disseminating findings internally where a possibility of learning from this evaluation to support similar current implementation; research could help similar project learning, lessons for country nationals such as Ministries;
  6. Prioritizing local capacity – Valuing Voices believes in using regional M&E capacity to do the work; where possible we partner with regional evaluators while also building capacity within our client's staff to carry out such work;
  7. Sharing and discussing findings locally: Valuing Voices believes knowledge learned needs to be shared in immediate feedback loops. We present: a) to each village after each site’s research; b) to local key partners and representatives from each village at the end of the qualitative Rapid Rural Appraisal findings; c) to the non-profit in-country at the end of qualitative research and d) internationally to headquarters at the end of the combined analysis of the qualitative and quantitative research with findings in the final report.

Project and Site Selection:

  1. Non-profit programming projects has been closed out for at least two years and no more than seven years (for recall);
  2. No other NGO has done very similar work in the region in the intervening years;
  3. The region selected is representative of the project as a whole (e.g. agro-ecological zones, economic/ livelihood/ health, educational or other sectoral criteria);
  4. Research areas are secure and safe (e.g. from civil unrest, severe drought/ floods, epidemics, to the degree possible);
  5. Timing does not interfere with urgent priorities of those involved with the study (e.g. livelihoods are not jeapordized in communities, holidays are kept, other technical work is not disrupted).

Resources (Time, Material and Project Expertise):

  1. Time: the research is qualitative followed by quantitative, coming to 80-90 days of qualitative and quantitative research overall (roughly 5 weeks of fieldwork in teams of 4-10, analysis, report-writing and presentation;
  2. Project and evaluation documents are available to inform and contextualize approach including activities, outcomes and projected impacts;
  3. Data: key to the fieldwork are village and participant lists from pre-closeout days so participants can be interviewed both during a Rapid Rural Appraisal and a follow-on household survey;
  4. Internal/external sectoral staff and at least one past project staff are part of the team to inform and ‘ground truth’ research;
  5. Logistics support is provided by the client; from vehicle/driver and lodging support in the field to materials such as mobile phones, flipcharts, photocopying and advances;
  6. A consultant or staff prepares the sites before the research teams come, e.g. to confirm communities are willing to be visited (each visit will be 2-5 days) and to identify participants and partners still there;
  7. Partners familiar with the closed project can be identified so they can be interviewed by the research team;
  8. Local language expertise is needed, e.g. translator to local language, as well as data entry personnel afterwards.
  9. While Valuing Voices provides the technical lead experts and statistical back office analysis, including sampling and rigorous analysis, senior non-profit staff are needed in-country for contextualization and input on preliminary findings, as well as senior technical staff to review the final product;
  10. A home for findings and on the road for dissemination: good knowledge management is needed for data retention, for the findings to have a sustainable ‘home’– be that info-graphics and print copies distributed to villages and partners or online repositories created that are language-accessible both nationally and by foreign donors; webinars, conference presentations etc are needed to optimize the learning via sustainability dissemination campaigns.

Much of this is needed for the research to be the best quality and yield the highest results. It is exciting learning is to be had from not only what communities and supporters could sustain but what they exceeded or dropped!  Consider doing one to see how post-project sustainability research can improve your current implementation, future design, and long-term self-sustainability!

IEG Blog Series Part II: Theory vs. Practice at the World Bank

by | Apr 7, 2015 | Beneficiaries, Estonia, Evaluation, ex-post evaluation, IEG, Local Participants, Niger, Nigeria, Participants, Participation, post-project evaluation, PPAR, self-sustainability, Sustainability, Sustainable development, Valuing Voices, World Bank

 

IEG Blog Series Part II: Theory vs. Practice at the World Bank

 

IEG logo

 

In Part I of this blog series, I described my research process for identifying the level to which the World Bank (WB) is conducting participatory post project sustainability evaluations for its many international development projects. Through extensive research and analysis of the WB’s IEG database, Valuing Voices concluded that there is a very loosely defined taxonomy for ex-post project evaluation at the WB, making it difficult to identify a consistent standard of evaluation methodology for sustainability impact assessments.

Particularly, we were concerned with identifying examples of direct beneficiary involvement in evaluating long-term sustainability outcomes, for instance by surveying/interviewing participants to determine which project objectives were self-sustained…and which were not. Unfortunately, it is quite rare for development organizations to conduct ex–post evaluations that involve all levels of project participants to contribute to long-term information feedback loops. However, there was one document type in the IEG database that gave us at Valuing Voices some room for optimism: Project Performance Assessment Reports (PPARs). PPARs are defined by the IEG as documents that are,

“…based on a review of the Implementation Completion Report (a self-evaluation by the responsible Bank department) and fieldwork conducted by OED [Operations Evaluation Department, synonymous with IEG]. To Prepare PPARs, staff examines project files and other documents, interview operation staff, and in most cases visit the borrowing country for onsite discussions with project staff and beneficiaries” [1].

The key takeaway from this definition is that these reports supplement desk studies (ICRs) with new fieldwork data provided, in part, by the participants themselves. The IEG database lists hundreds of PPAR documents, but I focused on only the 33 documents that came up when I queried “post-project”.

Here are a few commonalities to note about the 33 PPARs I studied:

  • They are all recent documents – the oldest document was published in 2004, and the most recent documents from 2014.
  • The original projects that are assessed in the PPARs were finalized anywhere from 2-10+ years before the PPAR was written, making them true ex-posts
  • They all claimed to involve mission site visits and communication with key project stakeholders, but they did not all claim to involve beneficiaries explicitly

 

Although the WB/IEG mentions that beneficiary participation takes place in “most” of the ex-post missions back to the project site in its definition of a PPAR, Valuing Voices was curious to know if there is a standard protocol for the level of participant involvement, the methods of data collection, and ultimately, the overall quality of the new fieldwork data collected to inform PPARs. For this data quality analysis, Valuing Voices identified these key criteria:

  • Overall summary of evaluation methods
  • Who was involved, specifically? Was there direct beneficiary participation? What were the research methods/procedures used?
  • What was the level of sustainability (termed Risk to Development Outcome* after 2006) established by the PPAR?
  • Was this different from the level of sustainability as projected by the preceding ICR report?
  • Were participants involved via interviews? (Yes/No)
  • If yes, were they semi-structured (open-ended questions allowing for greater variety/detail of qualitative data) or quantitative surveys
  • How many beneficiaries were interviewed/surveyed?
  • What % of total impacted beneficiary population was this number?
  • Was there a control group used? (Yes/No)

Despite our initial optimism, we determined that the quality of the data provided in these PPARs was highly variable, and overall quite low. A summary of the findings is as follows:

 

1. Rarely were ‘beneficiaries’ interviewed

  • Only 15% of the PPARs (5) gave details about the interview methodologies, but of this only 3% of the PPARs (1) described in detail how many participants were consulted, what they said and how they were interviewed (Nigeria 2014 [2]).
  • 54% of the reports (18), mentioned beneficiary input in data collected in the post-project mission, but gave no specific information on the number of participants involved nor were their voices cited nor was any information included on the methodologies used. The vast majority only vaguely referenced the findings of the post project mission, rather than data collection specifics. A typical example of this type of report is Estonia 2004 [1]
  • 30% of the PPARs (10) actually involved no direct participant/beneficiary participation in the evaluation process, with these missions only including stakeholders such as project staff, local government, NGOs, donors, consultants, etc.A typical example of this type of report is Niger 2005 [3]

These percentages are illustrated in Figure 1, below, which gives a visual breakdown of the number of reports that involved direct participant consultation with detailed methodologies provided (5), the number of reports where stakeholders were broadly consulted but no specific methodologies were provided (18), and the number of reports where no participants were directly involved in the evaluation process (10).

 

Graph 1

 

2. Sustainability of project outcomes was unclear

  • In 54% of cases, there was some change in the level of sustainability from the original level predicted in the ICR (which precedes and informs the PPAR) to the level established in the PPAR.  Ironically, of the 33 cases, 22 of them were classified as Likely or Highly Likely or Significantly Likely to be sustainable, yet participants were not asked for their input.
  • So on what basis was sustainability judged? Of the three cases where there was high participant consultation, the Nigerian project’s (where they asked 10% of participants for feedback) sustainability prospects was only moderate while India (also 10% feedback) and Kenya (14-20%) both were classified as likely to be sustainable.

Along the Y axis of Figure 2, below, is the spectrum of sustainability rankings observed in the PPARs, which range from “Negligible to Low” up to “High”. For each of the projects analyzed (there are 60 total projects accounted for in this graph, as some of the PPARs covered up to 4 individual projects in one report), the graph illustrates how many projects consulted participants, and how many failed to do so, for each evaluation outcome. As we can see, the majority of cases that were determined to be highly or significantly sustainable either did not consult participants directly or only consulted stakeholders broadly, with limited community input represented in the evaluation.  These are interesting findings, because although there is a lot of supposed sustainability being reported, very few cases actually involved the community participants in a meaningful way (to our knowledge, based on the lack of community consultation discussed in the reports). However, unless these evaluations are taking place at grassroots level, engaging the participants in a conversation about the true self-sustainability outcomes of projects, you can’t really know how sustainable the project is by only talking with donors, consultants, governments, etc. Are the right voices really being represented in this evaluation process? *Note: the “Sustainability” ranking was retitled “Risk to Development Outcomes” in 2006.

 

Graph 2

 

While projects were deemed sustainable, this is based on very little ‘beneficiary’ input. The significance of this information is simple: not enough is being done to ensure beneficiary participation in ALL STAGES of the development process, especially in the post-project time frame, even by prominent development institutions like the WB/IEG. While we commend the Bank for currently emphasizing citizen engagement via beneficiary feedback, this still seems to be more of a guiding theory than a habitualized practice [4]. Although all 34 documents I analyzed claimed there was “key stakeholder” or beneficiary participation, the reality is that no consistent procedural standard for eliciting such engagement could be identified.

Furthermore, the lack of specific details elaborating upon interview/survey methods, the number of participants involved, the discovery of any unintended outcomes, etc. creates a critical information void. As a free and public resource, the IEG database should not only be considered an important internal tool for the WB to catalog its numerous projects throughout time, but it is also an essential external tool for members of greater civil society who wish to benefit from the Bank’s extensive collection of resources – to learn from WB experiences and inform industry-wide best practices. 

For this reason, Valuing Voices implores the World Bank to step up its game and establish itself as a leader in post-project evaluation learning, not just in theory but also in practice. While these 33 PPARs represent just a small sample of the over 12,000 projects the WB has implemented since its inception, Valuing Voices hopes to see much more ex-post project evaluation happening in the future through IEG. Today we are seeing a decisive shift in the development world towards valuing sustainable outcomes over short-term fixes, towards informing future projects based on long-term data collection and learning, and towards community participation in all stages of the development process…

 

If one thing is certain, it is that global emphasis on sustainable development will not be going away anytime soon…but are we doing enough to ensure it?

 

Sources:

[1] World Bank OED. (2004, June 28). Project Performance Assessment Report: Republic of Estonia, Agriculture Project. Retrieved from http://documents.worldbank.org/curated/en/173891468752061273/pdf/295610EE.pdf

[2] World Bank OED. (2014, June 26). Project Performance Assessment Report: Nigeria, Second National Fadama Development Project. Retrieved from https://ieg.worldbankgroup.org/sites/default/files/Data/reports/Nigeria_Fadama2_PPAR_889580PPAR0P060IC0disclosed07070140_0.pdf

[3] World Bank OED. (2005, April 15). Project Performance Assessment Report: Niger, Energy Project. Retrieved from http://documents.worldbank.org/curated/en/899681468291380590/pdf/32149.pdf

[4] World Bank. (n.d.). Citizen Engagement: Incorporating Beneficiary Feedback in all projects by FY 18. Retrieved 2015, from https://web.archive.org/web/20150102233948/http://pdu.worldbank.org/sites/pdu2/en/about/PDU/EngageCitizens

 

When Funders Move On (Originally published by Stanford Social Innovation Review 03/15)

by | Mar 23, 2015 | Accountability, Aid effectiveness, Catholic Relief Services (CRS), Center for Effective Philanthropy, EU, ex-post evaluation, Hewlett Foundation, IATI, JICA, Keystone, OECD, post-project evaluation, SSIR, Sustainable development, USAID

When Funders Move On: Donors and nonprofits need to learn more about how to help program participants keep progressing after the support ends.

Imagine standing in Detroit or South-Central Los Angeles. A team of experts has come to help you out of grinding poverty. Some of these experts specialize in credit issues, others in education or health or gardening. They have funding for three years, so they set up offices, create participant lists and prioritize problems to tackle. They give you seeds and loans, and advice. And you—and others from your neighborhood—begin creating small businesses and home gardens. You and other adults learn about infant nutrition; children who live in your area get free school materials; teachers at your local schools receive extra training. Everyone begins to do better.

A year and a half passes. Another expert arrives to find out how things are going relative to the team’s projections. Some businesses are succeeding, others have faltered; some gardens are flourishing, others are neglected. You participate in a focus group, and you answer questions optimistically.

At the three-year mark, many of your neighbors are participating in this project, and tangible successes appear to be spreading. But suddenly, the experts are packing their boxes. The project office closes. The initiative has supposedly been “handed over” to the community. No one who worked for the project comes back.

 No one comes back. This is the state of affairs for too many so-called “sustainable international development” initiatives around the world, and it has to change.

As Gugelev and Stern brilliantly note in a recent SSIR article, we must be transparent about our “endgame”: Too many international development projects are bound to fixed endings and “fail to reckon with the gap between what the nonprofit can achieve and what the problem actually requires.” Due to fixed funding requirements, donors often leave when the calendar tells them to, whether or not a project has achieved the desired impact. And according to Valuing Voices research, they don’t even go back to assess the outcomes of their work or consider what (if anything) might help progress continue!

Since 2000, the US government has spent more than $280 billion on bilateral and multilateral assistance; the EU has spent $1.4 trillion. Just in 2002, US foundations, businesses, and NGOs spent more than $34 billion overseas. And while most taxpayers believe that this spending supports “sustainable development,” our research shows that 99 percent of the nonprofit grant and for-profit contracted projects these funds enabled were not evaluated after the funding concluded. Unfortunately, this continues: In 2014, the US spent $20 billion and the EU spent $80 billion on program assistance without any plan for post-project evaluation. This does not mean the projects are not sustainable; we simply do not know.

In fact, the United States Agency for International Development (USAID)—once considered the leader in post-project evaluations assessing relevance, effectiveness, efficiency, and sustainability—has managed only one post-project evaluation in 30 years (due later this year). And although thousands of documents appear in multilateral donor database searches as evaluations, most are “desk studies”—conducted remotely and not based on new fieldwork. Of these, only a few include feedback from program participants (leading the way are Japan’s International Cooperation Agency and the UN’s Organization for Economic Co-operation and Development, which have systematically done post-project evaluations).

ValuingVoicesArrowGraphic

There is usually terrific monitoring and evaluation during project implementation (red) and evaluation at start-mid-end (green), virtually no one returns afterwards (blue).

Why don’t we do a better job of following up? Are we afraid of the possibility of seeing poor results? If so, it’s time to face that fear. In some cases, we will surely see good results or even unanticipated positive impact—we’re missing that too.

Hewlett Foundation’s Fay Twersky implored nonprofits to “systematically solicit feedback from intended beneficiaries” in an SSIR podcast on Monitoring and Evaluation (M&E). We agree. We need to know more than whether those participants’ situations are improving while a program is in full swing, and we need to know what it will take for things to continue to improve after the funding goes away. Imagine the cost efficiencies we would gain by replicating activities that they could sustain. Imagine the cost-efficiencies and productivity if we prioritized activities with the largest sustainable return on investment (ROI). Now that is something impact investors could buy into.

There are positive signs on the horizon. According to Keystone Accountability, an increasing number of nonprofit organizations are committing to “making governments, NGOs, and donors more responsive to the needs of their constituents.” And funding that supports the idea of using participant feedback to improve programs and make them sustainable is on the rise, as Center for Effective Philanthropy’s “Hearing from Those We Seek to Help” and the Fund for Shared Insight have noted. All of this could yield a hugely different array of endgames that are sustainable by communities that can perhaps later even select development aid offers based on past effectiveness.

But there is still much to do. We offer the following recommendations to project implementers and donors, based on our own experience, and on our observations of several initiatives where we have seen project participants independently continuing and adapting work that was begun with external support:

  • Shift the development model from what donors and implementers think would be best to what the intended participants think is best. Design projects with them, and mandate that request for proposals (RFP) design involves communities.
  • Document, share, and discuss what was most sustained and what participating communities and local partners can do to sustain projects, and how we can sustainably support them through design and implementation to make them more effective.
  • Require a plan for transitioning to sustainability after projects close for any project that uses more than $1 million dollars. This should include handover plans to local nonprofits, with training and financial support; training for communities on how to manage the sustainable activities it prioritizes; financing mechanisms for those activities; and report sharing in IATI open-data format, with project data saved and stored in the cloud for global access.
  • Do post-project sustainability evaluations on all these projects, and discuss the results widely with other funders, the government, and the private sector, including how to feed back lessons into future design.
  • Advocate for participatory input in all evaluations. This input should make up 30 percent of future evaluation findings (now far less).
  • Consistently solicit feedback from local communities through national evaluators, both during and after projects, to better understand how the program you’re running or supporting from afar is working on the ground. Invest in building the national capacity and systems needed to make that feedback helpful for all stakeholders, including national governments.
  • Advocate for extensive civil society input into the United Nations’ Sustainable Development Goals so they serve our participants’ visions for the world they want.

 

As Peter Kimeu of Catholic Relief Services said to me, “It will be sustainable development if the people at community level are involved in designing and delivering their own dreams of development.”

 **************************************

Jindra Cekan (@WhatWeValue) is founder of Valuing Voices, with 28 years in international development design, monitoring, and evaluation. She has a doctorate from the Fletcher School of Law and Diplomacy; was a University of Cambridge Fellow; and works with foundation, nonprofit, and for-profit clients.

Are We Done Yet?

by | Mar 14, 2015 | Accountability, Aid effectiveness, Evaluation, Food for Peace (FFP), Food security, IATI, International aid, Local Participants, Maternal Child Health, MCHN, Open Data, Project design, Sustainability, Sustainable development

Are We Done Yet?

When are we off the hook, so to speak, for the well-being of the participants whom we said we'd make healthier, better fed, more educated, safer, etc?

 

America’s Agency for International Development (USAID) is the main channel for international development aid.  It is also an organization interested in learning from its programming and numerous contracts support such work. One such contract by FHI360/FANTA was Food for Peace tasking them to review the agency’s Title II development food aid from 2003-2009 covering 28 countries. This Second Food Aid and Food Security Assessment (FAFSA-2) Summary  found that such programs can “reduce undernutrition in young children, improve health and nutrition outcomes, and increase access to income and food” and also found practices that did not work well. 

 

While USAID has made enormous strides in the intervening six years on monitoring and evaluation (I was a consultant to USAID/PPL/LER from 2013-14), excellent recommendations that would support great, sustainable programs are unfulfilled:

Recommendations #1, 4 “USAID/FFP should develop an applied research agenda and sponsor studies that focus on the implementation of Title II programs in the field to better define what works and what does not…. [and] should select the review panel for new Title II applications… and give reviewers a ‘cheat sheet’ on interventions and approaches that USAID/FFP is and is not interested in funding because they work better or do not work as well, [and] provide this same information in the Request for Assistance” [Request for proposals].

 

Yes, all across our industry there is little learning from past evaluations for future design and Valuing Voices believes local participants and stakeholders need to be consulted to tell us what (still) works and what they want more of not only during implementation but long after. Their voices must support great design, as it’s their lives we go there to improve; they must be involved in the design of these original requests that non-profits design and fulfill. Further, the study found that only 1/3 of all evaluations were included in USAID’s database[1], and as Valuing Voices’ partner Sonjara has written in our blog, aid transparency requires data retention and access for learning to happen.

 

Recommendation #3 “USAID/FFP should include options for extensions of awards or separate follow-on awards to enable USAID/FFP to continue to support high-performing programs beyond five years and up to ten years… [as] longer implementation periods are associated with greater impact.”

 

This would address the ‘how much impact can we accomplish in 1, 3, 5 years” question that many of us in international non-profits ask ourselves. Finally, the graphic below is self-explanatory – USAID sees its role ending at close-out.

www_fsnnetwork_org_sites_default_files_fafsa2-summary-feb2013_pdf

The crux lies in their honest statement: "It was beyond the scope and resources of the FAFSA-2 to explore in any depth the sustainability of Title II development programs after they ended." While they state that there is merit in having impact while you intervene, such as "having a positive impact on the nutritional status of the first cohort of children is of immense benefit in its own right", they go on to say that "ideally, one would like to see mothers continuing positive child feeding practices and workers continuing to deliver services long after programs end… [yet] whether the [maternal child health and nutrition] interventions are sustainable beyond one generation is unknown and would require research."   This is because funding is pre-programmed, fixed to end within set 1, 3, 5 year increments, and no one goes back to learn how it all turned out.  This is what most needs to change, this illusion that what happens after closeout is no longer our issue, that the ‘positive impact’ we had while there is enough.

They are not alone. I think of NORAD, the Government of Norway's development arm as very progressive. So I went to  NORAD's website and searched for 'ex-post' (we do a lot of that at ValuingVoices). So like our World Bank blog on finding real ex-post evaluations, many many things are considered 'ex-post', including one actual evaluation in Palestine with fieldwork which asked participants and a few that looked at institutional sustainability. Many of the 100+ 'finds' were actually documents recommending ex-post. Typical of our other searches of other donors.  I emailed NORAD whether there were more with participant voices, yet they assured me they did them. Maybe our problem is in definitions and taxonomy again. Maybe we should call them post-project participant feedback?

Most of my colleagues would agree that the sustainability of activities aimed at making communities food secure in the long-term and independent of aid is a shared goal, one which short-term assistance aimed at huge impacts such as to ‘make communities food secure’ and ‘sustainably decrease malnutrition’ (common proposal goals) is unrealistic. We need participant voices to teach us how well we served them. We need to return, learn “what works and what does not”, and Value Voices in true sustained partnership. We all look forward to being done. 

 

[1] “Another major obstacle to transparency and learning from the Title II program experience was the fact that only one-third of the final evaluations were publicly available on the Development Experience Clearinghouse (DEC), despite the requirement that Awardees post them to the DEC…. [There was a lack of] cross-cutting studies or in-depth analyses of Title II evaluation results to advance organizational learning  [and] much greater use could be made of the evaluation data for systematic reviews, meta-analyses, secondary analyses, and learning.”

 

 

 

 

 

 

Can development projects be sustainable? Reblog from www.evaluace.com…

by | Mar 4, 2015 | Accountability, Aid effectiveness, Czech, Evidence-based policy, Mongolia, Sustainability, Sustainable development, Transparency

Mongolia

Can development projects be sustainable?

(Reblog http://www.evaluace.com by Inka Pibilova and Marie Korner)

 

What remained of Mongolian kindergartens

In 2012, we have ventured with my colleague Marie Koerner through steppes of Mongolia to learn what remained of the Czech-funded mobile kindergartens. People remembered a great project, nomadic women teaching in mobile kindergartens, children learning playfully in a safe environment and integrating better to primary schools… and their parents having more time for work or smaller siblings. Well, not much remained of it 4 months after extended project end. Despite an official agreement, the Mongolian Ministry of Education did not provide a budget. It rather accepted a bigger donation from the Asian Development  Bank for a similar project. Well, the new one did not benefit that many children in so many remote areas, but kindergartens had better qualified teachers, free meals and heating. And the donor was happy. Still, something remained of the Czech project – the positive attitude of the community towards pre-primary education and dedication to involve children at least in summer prep-schools organized by the government. Read the full evaluation report here in the Czech language.

The sustainability stories are diverse, yet factors are often similar. The devil is in the detail – for example, a dedicated village chief can make a big difference.

Sustainability or commercial continuity?

Based on different evaluations of Czech development projects conducted in 2010 to 2013, the Czech Ministry of Foreign Affairs started debating sustainability in 2013. Often, it was linked to commercial continuity – companies, ideally Czech, would continue to have business based on their previous successful engagement. Or at least income generation within the projects was highlighted. Questions arose if at least certain sectors, CSO or companies or types of projects are more sustainable than others, based on income generation or other factors.

What makes development projects sustainable?

Following the policy debate, the Czech platform of Civil Society Organisations – FoRS engaged Marie Koerner and me to analyse all evaluation reports, define what sustainability actually is and how it can be achieved. The key findings are below.

  1. Project sustainability is linked to continued benefits for intended beneficiariesafter donor funding has been withdrawn. Not all project activities need to continue. Not all benefits need to be seen as meaningful by the beneficiaries. Income generation is not always possible and does not necessarily contribute to poverty reduction. Especially if the income is generated by the Czech companies, not by the local actors, who are responsible for sustaining project benefits.
  2.  Sustainability of the Czech Development Cooperation projects varies. Review of 37 publicly available evaluation reports of Czech development projects shows no correlation between sustainability and specific sectors, contracting authorities, implementers or countries of implementation has not been detected. It is rather external and internal factors shown below which influence project sustainability.
  3. Sustainability factors are project specific and interdependent. They are external and internal. Often, even external factors can be influenced to achieve higher sustainability.
  4. Sustainability can be influenced throughout the project cycle. Each stakeholder plays a certain role at each phase of the project cycle.
  5. Good practice is available across sectors, including environment, agriculture, social development including education, economic development including energy as well as global development education and awareness raising. These sustainable Czech projects considered the key factors influencing sustainability in their plans and activities and thus contributed to lasting positive benefits for the recipients.

Overview of factors is displayed below:

Sustainability Factors

 

 

 

Sustainability Factors

 

 

 

 

Read the publication below, including different good practices. You can download ithere.

What next? What is your experience?

Based on the study, the Czech Development Agency together with FoRS organised an internal expert workshop on 13 March 2014 to debate possible steps for increasing the sustainability of Czech bilateral projects.

The results are yet to be seen. Feel free to share your resources and experience!

 

 

 

 

IEG Blog Series Part I: Pick a term, any term…but stick to it!

by | Mar 2, 2015 | Accountability, Beneficiaries, Evaluation, Evidence-based policy, ex-post evaluation, Feedback loops, impact evaluation, Local Participants, Participants, Participation, post-project evaluation, Results, self-sustainability, Sustainability, Sustainable development, Transparency, Uncategorized, Valuing Voices, World Bank

Pick a term, any term…but stick to it!

IEG logo

            Valuing Voices is interested in identifying learning leaders in international development that are using participatory post-project evaluation methods to learn about the sustainability of their development projects. These organizations not only believe they need to see the sustained impact of their projects by learning from what has worked and what hasn’t in the past, but also that participants are the most knowledgeable about such impacts. So how do they define sustainability? This is determined by asking questions such as the following: were project goals self-sustained by the ‘beneficiary’ communities that implemented these projects? By our VV definition, self-sustainability can only be determined by going back to the project site, 2-5 years after project closeout, to speak directly with the community about the long-term intended/unintended impacts. 

            Naturally, we turned to the World Bank (WB) – the world’s prominent development institution – to see if this powerhouse of development, both in terms of annual monetary investment and global breadth of influence, has effectively involved local communities in the evaluation of sustainable (or unsustainable) outcomes.  Specifically, my research was focused on identifying the degree to which participatory post-project evaluation was happening at the WB.

A fantastic blog* regarding participatory evaluation methods at the WB emphasizes the WB’s stated desire to improve development effectiveness by “ensuring all views are considered in participatory evaluation,” particularly through its community driven development projects. As Heider points out,

“The World Bank Group wants to improve its development effectiveness by, among others things, engaging citizens throughout the operational project cycle. It has set itself an ambitious target: 100% citizen engagement in projects that have clearly identifiable beneficiaries.”

Wow! Though these methods are clearly well intentioned, there seems to be a flaw in the terminology. The IEG says, “[Community driven development projects] are based on beneficiary participation from design through implementation, which make them a good example of citizen-centered assessment techniques in evaluation,” …however, this fails to recognize the importance of planning for community-driven post-project sustainability evaluations, to be conducted by the organization in order to collect valuable data concerning the long-term intended/unintended impacts of development work.

With the intention of identifying evidence of the above-mentioned mode of evaluation at the WB, my research process involved analyzing the resources provided by the WB’s Independent Evaluation Group (IEG) database of evaluations. As the accountability branch of the World Bank Group, the IEG works to gather institution-wide knowledge about the outcomes of the WBs finished projects. Its mission statement is as follows:

“The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group’s work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings.”

Another important function of the IEG database is to provide information for the public and external development organizations to access and learn from; this wealth of data and information about the World Bank’s findings is freely accessible online.

            When searching for evidence of post-project learning, I was surprised to find that the taxonomy varied greatly; e.g. projects I was looking for could be found under ‘post-project’, post project’, ‘ex-post’ or ‘ex post’. What was also unclear was any specific category under which these could be found, including a definition of what exactly is required in an IEG ex post impact evaluation. According to the IEG, there are 13 major evaluation categories, which are described in more detail here. I was expecting to find an explicit category dedicated to post-project sustainability, but instead this type of evaluation was included under Project Level Evaluations (which include PPARs and ICRs [Implementation Completion Reports]), and Impact evaluations.

This made it difficult to determine a clear procedural standard for documents reporting sustainability outcomes and other important data for the entire WB.

            I began my research process by simply querying a few key terms into the database. In the first step of my research, which will be elaborated upon in Part I in this blog series, I attempted to identify evidence of ex post sustainability evaluation at the IEG by searching for the term “post-project” in the database, which yielded 73 results when using a hyphen and 953 results without using a hyphen. I found it interesting the inconsistency in the number of results depending on the use of a hyphen, but in order to narrow the search parameters to conduct a manageable content analysis of the documents, I chose to breakdown these 73 results by document type to determine if there are any examples of primary fieldwork research. In these documents, the term “post-project” was not used in the title of the documents or referenced in the executive summary as the specific aim of the evaluation, but rather used to loosely define the ex post time frame. Figure 1 illustrates the breakdown of document types found in the sample of 73 documents that came up when I searched for the key term “post-project”:

            As the chart suggests, many of the documents (56% – which accounts for all of the pie chart slices except Project Level Evaluations) were purely desk studies – evaluating WB programs and the overall effectiveness of organization policies. These desk studies draw data from existing reports, such as those published at project closeout, without supplementing past data with new fieldwork research.

            Out of the 9 categories, the only document type that showed evidence of any follow up evaluations were the Project Performance Assessment Reports (PPARs), defined by the IEG as documents that are…

“…based on a review of the Implementation Completion Report (a self-evaluation by the responsible Bank department) and fieldwork conducted by OED [Operations Evaluation Department]. To prepare PPARs, OED staff examines project files and other documents, interview operational staff, and in most cases visit the borrowing country for onsite discussions with project staff and beneficiaries. The PPAR thereby seeks to validate and augment the information provided in the ICR, as well as examine issues of special interest to broader OED studies.”

            Bingo. This is what we’re looking for. The PPARs accounted for 32 out of the 73 results, or a total of 44%. As I examined the methodology used to conduct PPARs, I found that in the 32 cases that came up when I searched for “post-project”, after Bank funds were “fully dispersed to a project” and resources were withdrawn, the IEG sent a post-project mission back into the field to collaborate on new M&E with local stakeholders and beneficiaries. The IEG gathered new data through the use of field surveys or interviews to determine project effectiveness.

            Based on these findings, I conducted a supplementary search of the term “ex post”, which yielded 672 results. From this search, 11 documents were categorized by the IEG as “Impact Evaluations”, of which 3 showed evidence of talking with participants to evaluate for sustainability outcomes. In follow-up blogs in this series I will elaborate upon the significance of these additional findings and go into greater detail regarding the quality of the data in these 32 PPARs, but here are a few key takeaways from this preliminary research:

  • Taxonomy and definition of ex-post is missing. After committing approximately 15-20 hours of research time to this content analysis, it is clear that navigating the IEG database to search for methodology standards to evaluate for sustainability is a more complicated process than it should be for such a prominent learning institution. The vague taxonomy used to categorize post-project/ex-post evaluation by the WB limits the functionality of this resource as a public archive dedicated to informing the sustainability of development projects the World Bank has funded.
     
  • Despite affirmative evidence of participatory community involvement in the post-project evaluation of WB projects, not all PPARs in the IEG database demonstrated a uniform level of ‘beneficiary’ participation. In most cases, it was unclear how many community members impacted by the project were really involved in the ex-post process, which made it difficult to determine even a general range of the number of participants involved in post-project activity at the WB.
     
  • Although PPARs report findings based, in part, on post-project missions (as indicated in the preface of the reports), the specific methods/structure of the processes were not described, and oftentimes the participants were not explicitly referenced in the reports. (More detailed analysis on this topic to come in Blog Series Part 2!)
     
  • These surprisingly inconsistent approaches make it difficult to compare results across this evaluation type, as there is no precise status quo.

Finally, the World Bank, which has funded 12,000 projects since its inception, should have far more than 73 post-project/ ex-post evaluations…but maybe I’m just quibbling with terms.

Stay tuned for PART II of this series, coming soon!