by Kelsey Lopez | Apr 15, 2014 | Bilateral organizations, Evaluation, International aid, JICA, Results, Sustainability, Sustainable development, Uncategorized, World Bank
The lack of ex-post project evaluation at the World Bank: One has no power
The World Bank has a huge repository of 8,483 evaluation resources in its e-library database, so naturally Valuing Voices was very interested in investigating how many of those resources were ex-post evaluations of past World Bank projects. After searching the term “ex post evaluation” in the e-library, I ended up with 260 hits from those initial 8,483 resources that were a match. This looked like great news to have so many potential ex-post evaluations to analyze from such a powerhouse in international development as the World Bank. From the initial 260 hits, I expected about 50 of them to be what we consider to be ‘true’ ex-post, which is an evaluation conducted a few years after a project has been completed to assess for factors such as sustainability and long-term effectiveness of the program after donor resources had been withdrawn.
However, when I began to sift through the resources in more detail, the results were not exactly as we had anticipated. In order to determine how many of the 260 “ex-post evaluation” hits were true ex-post the process was simple, albeit time consuming. I looked through every hit, reading the abstracts provided by the World Bank and investigating individual resources in more detail if they seemed promising. While doing this, I categorized each hit by document type, keeping a tally of all the totals. The results were as follows:
|
Document Type
|
Number Encountered
|
|
Impact Evaluations
|
21
|
|
Retrospective Evaluations
|
21
|
|
Non-Evaluations (literature review, recommendations, guidelines, etc. related to evaluations)
|
53
|
|
Other (Policy reports, annual reports, sourcebooks, etc. not related to evaluations)
|
164
|
|
Ex-Post Evaluations
|
1
|
|
|
Total: 260
|
Did anything about these results surprise you? Yes, you read that right. Out of all 260 hits that came up from the search “ex-post evaluation” in the World Bank online database, an astounding grand total of one was a true ex-post evaluation of a past project. A bilateral counterpart, Japan’s JICA, has done 236 in 2009-2011 on past ODA projects, one of our rare stars in ex-post learning.
Suffice it to say, Valuing Voices was shocked by these results. There exists a clear need, based on this research, for the World Bank to contribute to the process of informing future projects by learning from past experiences, successes, mistakes, and community feedback through the valuable ex-post evaluation method. While impact and retrospective evaluations are indeed important, the nature of compiling many evaluations into one broad analysis doesn’t allow for a detailed assessment of how individual projects performed, especially when the respondents in many other multilateral ex-posts tend to remain government counterparts rather than local respondents. This type of comprehensive analysis of the long-term sustainability of completed projects can only be done by conducting ex-post evaluations for projects on a case-by-case basis.
Jindra Cekan (head of ValuingVoices) was invited to attend and speak at the World Bank’s Civil Society Organization spring meetings last week, and, armed with my findings, asked why we don’t see ex-posts at two sessions. Astrid Marroh, a senior staffer tasked with setting new strategy for the Bank, answered that longitudinal learning is, “a nut we have not yet cracked”. Varun Gauri, writing the major World Development Report 2015 on Mind and Culture at the Bank, said that not only do Bank staff need to, “change the incentives from managing projects as managers to focus on the project’s ultimate aim,” but also that the Bank, “needs to follow the private sector’s approach by ‘Dog-fooding’ our projects– living our own projects“ (where private sector producers try and eat the dog food themselves).
So what is the takeaway lesson learned form all this? Organizations like the World Bank are what set the precedent in international development, yet even this influential international organization fails to conduct regular ex-post evaluations. Despite having plentiful literature and recommendations on how to conduct ex-post evaluations and why they are important to the development process, it is clear that ex-post is not happening at the World Bank. Now is time for the organization to change the status quo and start valuing the voices of their project participants by conducting rigorous ex-post evaluations of their projects including feedback from the community level, in order to finally address this deficiency and establish a cycle of feedback loops and informed decision making that will benefit all involved – and make ‘development aid’ obsolete.
by Siobhan Green | Apr 7, 2014 | Participation, Sustainability, Sustainable development, Uncategorized
What to consider for sustainability….
Recently I read an article on “how to make a computer lab sustainable”. I read it with interest because I find the question about sustainability in international development fascinating. What do we mean by sustainability? Every project has a requirement to be sustainable, but the definition seems to vary based on the project or the donor. Does it mean “continue to exist beyond the period of performance?” “Positive impacts continue beyond the period of performance?”, “Not just continue to exist but grow and scale?”
Do we mean sustainability of operations? Of financial independence? Impact? Involvement of local populations? All of the above?
In the article I mentioned, sustainability really meant the hardware and software would continue beyond the funding cycle and that they were looking at fee for service opportunities/integration into an existing organization. But they didn’t mention any details about the business model –would the fee for service cover the increased costs for internet, hardware/software support and purchases, salaries of staff, electricity? Would they need to find additional sponsorship from donors or partner organizations? Do they have a management team which is capable of managing the budget, of ensuring high quality of services to customers, of finding new sponsors? What about growth and changes in technology infrastructure?
Now, the project may well have included these aspects in their design, but the article didn’t mention any of these things. And for me, sustainability starts with thinking about treating the seed funding you get as an initial investment, which should lead to a self-sustaining model that won’t require more investment (at least not for the same types of activities). If the model involves transitioning to local staff or partner organization, that needs to be both explicit and planned from the start, with explicit assumptions and expectations on what sustainability means.
Because otherwise sustainability is kind of a meaningless term.
Who should decide what sustainability is unless we Value all Voices to discuss and decide? What are your thoughts?
by Jindra Cekan | Mar 26, 2014 | Accountability, Africa, Bilateral organizations, Credit, Donors, Ethiopia, Evaluation, Food security, International aid, International non-profits, LIvelihoods, Local Participants, MDG, Milenium Development Goals, NGO, Participation, Results, Rural Development, Sustainability, Sustainable development, Transparency, Women and gender
Stepping up community self-sustainability, one [Ethiopian] step at a time
Having just come back from evaluation and design fieldwork for an Ethiopian Red Cross (ERCS)/ Swedish Red Cross/ Federation of the Red Cross and Red Crescent project, the power of communities is still palpable in my mind. They know what great impact looks like. They know what activities they can best sustain themselves. It’s up to us to ask, listen and learn from them and support their own monitoring/ evaluating/ reporting. It’s up to us to share such learning with others and to act on it everywhere.
There are a myriad of possible sustainability indicators, and the outcome indicators below, suggested by 116 rural participants from Tigray, Ethiopia seem to fall into two categories of expected changes: Assets and Life Quality (Table 1). As the food security/ livelihood project extended credit for animal purchases, it is logical that tracking increased income, savings, assets, and home investments plus expenditures on food and electricity appeared.
We gleaned this from discussions with participants, asking them “what can we track together that would show that we had impact”? Our question led to a spirited discussion of not only what was traceable, but also what could be publicly posted and ‘ground-truthed’ by the community. Discussing indicators led to even deeper conversations about the causes of food insecurity which were illuminating to staff. What was surprising, for instance, was the extent to which families saw changing seasonal child-field labor practices in favor of 100% child-school attendance as great indicators. School attendance (or lack thereof) was dependent on families’ need for children’s seasonal labor in the fields. Community members said they knew who sent their children or not, which no only ‘cleaned’ the publicly posted data but triangulated implementer surveys and opened room for discussions of vulnerability.
Not only is this exciting for the project’s outcome tracking but even more importantly, our team proposed to create a community self-monitoring system, suggested in by Causemann/Gohl in an IIED PLA Notes article– “Tools for measuring change: self-assessment by communities” used in Africa and Asia. This learning, management and reporting process will fill a gaping need as current “monitoring systems serve only for donor accountability, but neither add value for poor people nor for the implementing NGOs because they do not improve effectiveness on the ground.” The authors found that not only “participatory data collection produces higher quality data in some fields than standard extractive methodologies [as] understanding the context leads to a higher accuracy of data and learning processes [which] increase the level of accountability… “ but also that such shared collaboration builds mutual learning and bridge-building.” While our community members may have offered to track this publicly to make this partner happy, men and women discussed this excitedly and embraced the idea of self-monitoring happily. ERCS will be discussing with communities to either track data monthly in notebooks or on a large chart hung in the woreda office for transparency. Data (Chart 1) would include these asset and quality of life indicators as well as loan repayments (tracked vertically) while households (tracked horizontally) could see who was meeting the goal (checked boxes), not meeting it fully (dashed boxes) or not meeting it at all yet (blank boxes). Community members corrected each other as they devised the indicators during our participatory research and this openness reassures us that the public monitoring will be quite transparent as well.
[1]
Further, what was especially satisfying was getting feedback from across the three tibias (sub-regions) on what activities they felt they could sustain themselves irrespective of the project’s continuation. Table 2 shows us which activities communities felt were most self-sustainable by households; these could form the core of the follow on project. Sheet/goats, poultry and oxen for fattening were highly prioritized by both women and men, in addition to a few choosing improved dairy cows. The convergence of similar responses was gratifying and somewhat unexpected, as there were several other project activities. The communities’ own priorities need to be seriously considered as currently they get only one loan per family and thus self-sustainable activities are key.
There is more to incorporate in future project planning by NGOs like Ethiopia’s ERCS. The NGO-IDEAs concept mentioned above also includes involving project participants in setting goals and targets themselves, differentiating between who achieved them and why, and brainstorming who/what contributes to it and what they should do next. Peer groups, development agencies and any actors could collect and learn from the data. Imagine the empowerment were communities to design, monitor and evaluate and tell us as their audience!
And they must, according to ODI UK’s Watkins, who has a clear vision on how to achieve a global equity agenda for the post-2015 MDG goals. He suggests converting the principle of ‘leave no one behind’ into measurable targets. He argues that, by introducing a series of ‘stepping stone’ benchmarks, the world can set ambitious goals on equity by 2030. He writes, wisely, that “narrowing these equity deficits is not just an ethical imperative but a condition for accelerated progress towards the ambitious 2030 targets. There are no policy blueprints. However, the toolkit for governments actively seeking to narrow disparities …has to include some key elements [such as] identifying who is being left behind and why is an obvious starting point. That’s why improvements to the quality of data available to policy-makers is an equity issue in its own right”. Valuing Voices believes who creates that data is an equally compelling equity issue.
So how will we reach these ambitious targets by 2030? By putting in stepping stone targets, returning project design functions to the ultimate clients – the communities themselves- and matching their wants with what we long to transfer to them. In this way we will be Valuing their Voices so much that they evaluate our projects jointly and we can respond. That’s how it should always have been.
What are your thoughts on this? We long to know.
Sources:
[1] Ashley, H., Kenton, N., & Milligan, A. (Eds.). (2013). Tools for supporting sustainable natural resource management and livelihoods. Participatory Learning and Action, (66). Retrieved from https://pubs.iied.org/14620IIED/
[2] Watkins, K. (2013, October 17). Leaving no-one behind: An equity agenda for the post- 2015 goals. Retrieved from https://www.odi.org/blogs/7924-leaving-no-one-behind-equity-agenda-post-2015-goals
