by Kelsey Lopez | Apr 7, 2015 | Beneficiaries, Estonia, Evaluation, ex-post evaluation, IEG, Local Participants, Niger, Nigeria, Participants, Participation, post-project evaluation, PPAR, self-sustainability, Sustainability, Sustainable development, Valuing Voices, World Bank
IEG Blog Series Part II: Theory vs. Practice at the World Bank
In Part I of this blog series, I described my research process for identifying the level to which the World Bank (WB) is conducting participatory post project sustainability evaluations for its many international development projects. Through extensive research and analysis of the WB’s IEG database, Valuing Voices concluded that there is a very loosely defined taxonomy for ex-post project evaluation at the WB, making it difficult to identify a consistent standard of evaluation methodology for sustainability impact assessments.
Particularly, we were concerned with identifying examples of direct beneficiary involvement in evaluating long-term sustainability outcomes, for instance by surveying/interviewing participants to determine which project objectives were self-sustained…and which were not. Unfortunately, it is quite rare for development organizations to conduct ex–post evaluations that involve all levels of project participants to contribute to long-term information feedback loops. However, there was one document type in the IEG database that gave us at Valuing Voices some room for optimism: Project Performance Assessment Reports (PPARs). PPARs are defined by the IEG as documents that are,
“…based on a review of the Implementation Completion Report (a self-evaluation by the responsible Bank department) and fieldwork conducted by OED [Operations Evaluation Department, synonymous with IEG]. To Prepare PPARs, staff examines project files and other documents, interview operation staff, and in most cases visit the borrowing country for onsite discussions with project staff and beneficiaries” [1].
The key takeaway from this definition is that these reports supplement desk studies (ICRs) with new fieldwork data provided, in part, by the participants themselves. The IEG database lists hundreds of PPAR documents, but I focused on only the 33 documents that came up when I queried “post-project”.
Here are a few commonalities to note about the 33 PPARs I studied:
- They are all recent documents – the oldest document was published in 2004, and the most recent documents from 2014.
- The original projects that are assessed in the PPARs were finalized anywhere from 2-10+ years before the PPAR was written, making them true ex-posts
- They all claimed to involve mission site visits and communication with key project stakeholders, but they did not all claim to involve beneficiaries explicitly
Although the WB/IEG mentions that beneficiary participation takes place in “most” of the ex-post missions back to the project site in its definition of a PPAR, Valuing Voices was curious to know if there is a standard protocol for the level of participant involvement, the methods of data collection, and ultimately, the overall quality of the new fieldwork data collected to inform PPARs. For this data quality analysis, Valuing Voices identified these key criteria:
- Overall summary of evaluation methods
- Who was involved, specifically? Was there direct beneficiary participation? What were the research methods/procedures used?
- What was the level of sustainability (termed Risk to Development Outcome* after 2006) established by the PPAR?
- Was this different from the level of sustainability as projected by the preceding ICR report?
- Were participants involved via interviews? (Yes/No)
- If yes, were they semi-structured (open-ended questions allowing for greater variety/detail of qualitative data) or quantitative surveys
- How many beneficiaries were interviewed/surveyed?
- What % of total impacted beneficiary population was this number?
- Was there a control group used? (Yes/No)
Despite our initial optimism, we determined that the quality of the data provided in these PPARs was highly variable, and overall quite low. A summary of the findings is as follows:
1. Rarely were ‘beneficiaries’ interviewed
- Only 15% of the PPARs (5) gave details about the interview methodologies, but of this only 3% of the PPARs (1) described in detail how many participants were consulted, what they said and how they were interviewed (Nigeria 2014 [2]).
- 54% of the reports (18), mentioned beneficiary input in data collected in the post-project mission, but gave no specific information on the number of participants involved nor were their voices cited nor was any information included on the methodologies used. The vast majority only vaguely referenced the findings of the post project mission, rather than data collection specifics. A typical example of this type of report is Estonia 2004 [1]
- 30% of the PPARs (10) actually involved no direct participant/beneficiary participation in the evaluation process, with these missions only including stakeholders such as project staff, local government, NGOs, donors, consultants, etc.A typical example of this type of report is Niger 2005 [3]
These percentages are illustrated in Figure 1, below, which gives a visual breakdown of the number of reports that involved direct participant consultation with detailed methodologies provided (5), the number of reports where stakeholders were broadly consulted but no specific methodologies were provided (18), and the number of reports where no participants were directly involved in the evaluation process (10).
2. Sustainability of project outcomes was unclear
- In 54% of cases, there was some change in the level of sustainability from the original level predicted in the ICR (which precedes and informs the PPAR) to the level established in the PPAR. Ironically, of the 33 cases, 22 of them were classified as Likely or Highly Likely or Significantly Likely to be sustainable, yet participants were not asked for their input.
- So on what basis was sustainability judged? Of the three cases where there was high participant consultation, the Nigerian project’s (where they asked 10% of participants for feedback) sustainability prospects was only moderate while India (also 10% feedback) and Kenya (14-20%) both were classified as likely to be sustainable.
Along the Y axis of Figure 2, below, is the spectrum of sustainability rankings observed in the PPARs, which range from “Negligible to Low” up to “High”. For each of the projects analyzed (there are 60 total projects accounted for in this graph, as some of the PPARs covered up to 4 individual projects in one report), the graph illustrates how many projects consulted participants, and how many failed to do so, for each evaluation outcome. As we can see, the majority of cases that were determined to be highly or significantly sustainable either did not consult participants directly or only consulted stakeholders broadly, with limited community input represented in the evaluation. These are interesting findings, because although there is a lot of supposed sustainability being reported, very few cases actually involved the community participants in a meaningful way (to our knowledge, based on the lack of community consultation discussed in the reports). However, unless these evaluations are taking place at grassroots level, engaging the participants in a conversation about the true self-sustainability outcomes of projects, you can’t really know how sustainable the project is by only talking with donors, consultants, governments, etc. Are the right voices really being represented in this evaluation process? *Note: the “Sustainability” ranking was retitled “Risk to Development Outcomes” in 2006.
While projects were deemed sustainable, this is based on very little ‘beneficiary’ input. The significance of this information is simple: not enough is being done to ensure beneficiary participation in ALL STAGES of the development process, especially in the post-project time frame, even by prominent development institutions like the WB/IEG. While we commend the Bank for currently emphasizing citizen engagement via beneficiary feedback, this still seems to be more of a guiding theory than a habitualized practice [4]. Although all 34 documents I analyzed claimed there was “key stakeholder” or beneficiary participation, the reality is that no consistent procedural standard for eliciting such engagement could be identified.
Furthermore, the lack of specific details elaborating upon interview/survey methods, the number of participants involved, the discovery of any unintended outcomes, etc. creates a critical information void. As a free and public resource, the IEG database should not only be considered an important internal tool for the WB to catalog its numerous projects throughout time, but it is also an essential external tool for members of greater civil society who wish to benefit from the Bank’s extensive collection of resources – to learn from WB experiences and inform industry-wide best practices.
For this reason, Valuing Voices implores the World Bank to step up its game and establish itself as a leader in post-project evaluation learning, not just in theory but also in practice. While these 33 PPARs represent just a small sample of the over 12,000 projects the WB has implemented since its inception, Valuing Voices hopes to see much more ex-post project evaluation happening in the future through IEG. Today we are seeing a decisive shift in the development world towards valuing sustainable outcomes over short-term fixes, towards informing future projects based on long-term data collection and learning, and towards community participation in all stages of the development process…
If one thing is certain, it is that global emphasis on sustainable development will not be going away anytime soon…but are we doing enough to ensure it?
Sources:
[1] World Bank OED. (2004, June 28). Project Performance Assessment Report: Republic of Estonia, Agriculture Project. Retrieved from http://documents.worldbank.org/curated/en/173891468752061273/pdf/295610EE.pdf
[2] World Bank OED. (2014, June 26). Project Performance Assessment Report: Nigeria, Second National Fadama Development Project. Retrieved from https://ieg.worldbankgroup.org/sites/default/files/Data/reports/Nigeria_Fadama2_PPAR_889580PPAR0P060IC0disclosed07070140_0.pdf
[3] World Bank OED. (2005, April 15). Project Performance Assessment Report: Niger, Energy Project. Retrieved from http://documents.worldbank.org/curated/en/899681468291380590/pdf/32149.pdf
[4] World Bank. (n.d.). Citizen Engagement: Incorporating Beneficiary Feedback in all projects by FY 18. Retrieved 2015, from https://web.archive.org/web/20150102233948/http://pdu.worldbank.org/sites/pdu2/en/about/PDU/EngageCitizens
by Kelsey Lopez | Mar 2, 2015 | Accountability, Beneficiaries, Evaluation, Evidence-based policy, ex-post evaluation, Feedback loops, impact evaluation, Local Participants, Participants, Participation, post-project evaluation, Results, self-sustainability, Sustainability, Sustainable development, Transparency, Uncategorized, Valuing Voices, World Bank
Pick a term, any term…but stick to it!
Valuing Voices is interested in identifying learning leaders in international development that are using participatory post-project evaluation methods to learn about the sustainability of their development projects. These organizations not only believe they need to see the sustained impact of their projects by learning from what has worked and what hasn’t in the past, but also that participants are the most knowledgeable about such impacts. So how do they define sustainability? This is determined by asking questions such as the following: were project goals self-sustained by the ‘beneficiary’ communities that implemented these projects? By our VV definition, self-sustainability can only be determined by going back to the project site, 2-5 years after project closeout, to speak directly with the community about the long-term intended/unintended impacts.
Naturally, we turned to the World Bank (WB) – the world’s prominent development institution – to see if this powerhouse of development, both in terms of annual monetary investment and global breadth of influence, has effectively involved local communities in the evaluation of sustainable (or unsustainable) outcomes. Specifically, my research was focused on identifying the degree to which participatory post-project evaluation was happening at the WB.
A fantastic blog* regarding participatory evaluation methods at the WB emphasizes the WB’s stated desire to improve development effectiveness by “ensuring all views are considered in participatory evaluation,” particularly through its community driven development projects. As Heider points out,
“The World Bank Group wants to improve its development effectiveness by, among others things, engaging citizens throughout the operational project cycle. It has set itself an ambitious target: 100% citizen engagement in projects that have clearly identifiable beneficiaries.”
Wow! Though these methods are clearly well intentioned, there seems to be a flaw in the terminology. The IEG says, “[Community driven development projects] are based on beneficiary participation from design through implementation, which make them a good example of citizen-centered assessment techniques in evaluation,” …however, this fails to recognize the importance of planning for community-driven post-project sustainability evaluations, to be conducted by the organization in order to collect valuable data concerning the long-term intended/unintended impacts of development work.
With the intention of identifying evidence of the above-mentioned mode of evaluation at the WB, my research process involved analyzing the resources provided by the WB’s Independent Evaluation Group (IEG) database of evaluations. As the accountability branch of the World Bank Group, the IEG works to gather institution-wide knowledge about the outcomes of the WBs finished projects. Its mission statement is as follows:
“The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group’s work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings.”
Another important function of the IEG database is to provide information for the public and external development organizations to access and learn from; this wealth of data and information about the World Bank’s findings is freely accessible online.
When searching for evidence of post-project learning, I was surprised to find that the taxonomy varied greatly; e.g. projects I was looking for could be found under ‘post-project’, post project’, ‘ex-post’ or ‘ex post’. What was also unclear was any specific category under which these could be found, including a definition of what exactly is required in an IEG ex post impact evaluation. According to the IEG, there are 13 major evaluation categories, which are described in more detail here. I was expecting to find an explicit category dedicated to post-project sustainability, but instead this type of evaluation was included under Project Level Evaluations (which include PPARs and ICRs [Implementation Completion Reports]), and Impact evaluations.
This made it difficult to determine a clear procedural standard for documents reporting sustainability outcomes and other important data for the entire WB.
I began my research process by simply querying a few key terms into the database. In the first step of my research, which will be elaborated upon in Part I in this blog series, I attempted to identify evidence of ex post sustainability evaluation at the IEG by searching for the term “post-project” in the database, which yielded 73 results when using a hyphen and 953 results without using a hyphen. I found it interesting the inconsistency in the number of results depending on the use of a hyphen, but in order to narrow the search parameters to conduct a manageable content analysis of the documents, I chose to breakdown these 73 results by document type to determine if there are any examples of primary fieldwork research. In these documents, the term “post-project” was not used in the title of the documents or referenced in the executive summary as the specific aim of the evaluation, but rather used to loosely define the ex post time frame. Figure 1 illustrates the breakdown of document types found in the sample of 73 documents that came up when I searched for the key term “post-project”:
Figure 1: Breakdown by Document Type out of Total 73 Results when searching post-project
As the chart suggests, many of the documents (56% – which accounts for all of the pie chart slices except Project Level Evaluations) were purely desk studies – evaluating WB programs and the overall effectiveness of organization policies. These desk studies draw data from existing reports, such as those published at project closeout, without supplementing past data with new fieldwork research.
Out of the 9 categories, the only document type that showed evidence of any follow up evaluations were the Project Performance Assessment Reports (PPARs), defined by the IEG as documents that are…
“…based on a review of the Implementation Completion Report (a self-evaluation by the responsible Bank department) and fieldwork conducted by OED [Operations Evaluation Department]. To prepare PPARs, OED staff examines project files and other documents, interview operational staff, and in most cases visit the borrowing country for onsite discussions with project staff and beneficiaries. The PPAR thereby seeks to validate and augment the information provided in the ICR, as well as examine issues of special interest to broader OED studies.”
Bingo. This is what we’re looking for. The PPARs accounted for 32 out of the 73 results, or a total of 44%. As I examined the methodology used to conduct PPARs, I found that in the 32 cases that came up when I searched for “post-project”, after Bank funds were “fully dispersed to a project” and resources were withdrawn, the IEG sent a post-project mission back into the field to collaborate on new M&E with local stakeholders and beneficiaries. The IEG gathered new data through the use of field surveys or interviews to determine project effectiveness.
Based on these findings, I conducted a supplementary search of the term “ex post”, which yielded 672 results. From this search, 11 documents were categorized by the IEG as “Impact Evaluations”, of which 3 showed evidence of talking with participants to evaluate for sustainability outcomes. In follow-up blogs in this series I will elaborate upon the significance of these additional findings and go into greater detail regarding the quality of the data in these 32 PPARs, but here are a few key takeaways from this preliminary research:
-
Taxonomy and definition of ex-post is missing. After committing approximately 15-20 hours of research time to this content analysis, it is clear that navigating the IEG database to search for methodology standards to evaluate for sustainability is a more complicated process than it should be for such a prominent learning institution. The vague taxonomy used to categorize post-project/ex-post evaluation by the WB limits the functionality of this resource as a public archive dedicated to informing the sustainability of development projects the World Bank has funded.
-
Despite affirmative evidence of participatory community involvement in the post-project evaluation of WB projects, not all PPARs in the IEG database demonstrated a uniform level of ‘beneficiary’ participation. In most cases, it was unclear how many community members impacted by the project were really involved in the ex-post process, which made it difficult to determine even a general range of the number of participants involved in post-project activity at the WB.
-
Although PPARs report findings based, in part, on post-project missions (as indicated in the preface of the reports), the specific methods/structure of the processes were not described, and oftentimes the participants were not explicitly referenced in the reports. (More detailed analysis on this topic to come in Blog Series Part 2!)
-
These surprisingly inconsistent approaches make it difficult to compare results across this evaluation type, as there is no precise status quo.
Finally, the World Bank, which has funded 12,000 projects since its inception, should have far more than 73 post-project/ ex-post evaluations…but maybe I’m just quibbling with terms.
Stay tuned for PART II of this series, coming soon!
by Kelsey Lopez | May 21, 2014 | Accountability, Africa, Governance, ICT, ICT4D, International aid, Literacy, mobiles, Nigeria, Participation, Youth
Youth Series Part III: The Role of ICT4D (Information and Communications Technology for Development) in Empowering Youth
Youth Series Part I here Factors Hindering Youth Participation in Development
Youth Series Part II here How Technology Enables Youth Participation
For international multilateral organizations that are funding billions of dollars annually for a wide range of initiatives aimed at improving the socioeconomic conditions in developing countries, the challenge these organizations should undertake is to ensure a youth-centric focus within their funding for programs. According to the Pan-African Youth Union, youth empowerment is defined as, “a structured process where young people gain the ability and authority to make real economic, social and political decisions. [They] believe this is the process that builds capacity to implement change, in young people, for use in their own lives, their communities, and in their society, by acting on issues that they define as important.”
A key takeaway is that problems and solutions are best addressed when they are self-defined. The problem that has been most adamantly professed revolves around employment and educational opportunities for so many youth who feel that they are not adequately prepared for the demands of the modern labor market. As a result, we propose that international development organizations fill the institutional void that exists in many developing countries by focusing their programming on solving problems such as poverty, unemployment and education with what has also been identified as an empowering tool in the modern era: technology. The most comprehensive solution that involves all of these aspects is greater Information and Communications Technology (ICT) skills training, listening, and using this medium as one method for combating the development challenges youth face in today’s world.
By orienting many youth development initiatives towards ICT skills, there is no doubt that developing youth with have much greater advantages to propel their capacity to be prosperous members of society. This is because, “equitable access to information, knowledge (or know-how) and education is one of the most vital principles in the emerging global knowledge economy. ICTs are practical tools in narrowing knowledge gaps between countries, regions and also people by providing new frontiers in the areas of information exchange, intellectual freedom and online education.” Additionally, “in the knowledge era continuous education and training is the only way for job security, especially if the education and training is in ICT-related skills.” The role of international development organizations should be to enable this type of progressive skills training both just for such access but also as a means for listening to our clients through mobile, Facebook, Yammer, Twitter and other applications. Also by improving access to ICT education programs to youth cohorts, they are more competitive in a global market that is increasingly demanding of workers with advanced ICT skills. These programs must tackle the “widening digital divide” between developed and developing countries to ensure a more sustainable and balanced development scheme.
To this point, the Executive Secretary of the United Nation Economic Commission for Africa (UNECA), Mr. Abdoulie Janneh, gave a statement at the 2011 African Press Organization (APO) forum themed ‘Accelerating Youth Empowerment for Sustainable Development’, which highlighted the fact that human capital is key in facilitating growth, and with greater education and training the African youth can contribute more to development and growth for the continent. Nonetheless, he could not go without saying that, “several commitments, policies and programmes on youth education and employment have been prioritized at national, sub-regional and global levels to improve the livelihoods of young people in Africa. However, these initiatives have yet to translate into the desired outcomes. Thus, concerted and innovative efforts are still required especially at a time when the youth population continues to increase.” Again we see the trend that current policies have thus far failed to provide the circumstances necessary for youth empowerment to become realized in many African counties, which means development is happening too slowly for the millions of African youths who could be contributing invaluable skills to their societies – if they only had the means- and we were listening and funding their priorities!
An example of a good ICT4D training program is the Youth Empowerment Program (YEP) in Nigeria, which was a two-year program implemented by the International Youth Foundation (IYF) and Microsoft, to “to improve the employability of disadvantaged African youth in Nigeria between ages 16 to 35. The program, with support from Microsoft, worked with LEAP Africa and local partners to provide demand-driven training in information and communications technology (ICT), life skills, entrepreneurship and employment services.” Over two years, the program addressed the inadequacy of technical skills and lack of labor market information in the Nigerian youth by providing training to “improve the employability prospects of 2,500 young people throughout the country,” in an aim to place 70% of the program participants in jobs, internships, self-employment or community service opportunities with greater capacity in education and training. Six months after project completion, the project was evaluated by interviewing a sample follow-up cohort of 69 participants:
· “All together, 55% of the respondents were employed, self-employed, participated in an internship or community service, or continued their studies after the training.” (This number is thought to be low, primarily because of the few employment opportunities in Bauchi, where the follow-up participants were from. This is typical in many cities where demand far outstrips employment opportunities)
· “Over 78% of the respondents in the sample follow-up cohort confirmed that the ICT training had improved their employment prospects.” They indicated that this was because ICT skills are important selection criteria in the job market,” and there was also a significant increase in the follow-up cohort’s use of computers.
Unfortunately there was no data on employment that was using these new ICT skills; more data is needed to compare those trained versus untrained regarding employment using these skills used, and how much more ‘development’ was fostered by such trainings. Yet given our dependence on technology, technical illiteracy seems a logical barrier. IYF has identified eight high-growth sectors for ICT-enabled youth employment, in fields such as, “Banking and Financial Services, Telecommunications, Information Technology, Oil and Gas, Education and Training, Media, Marketing and Advertising, Hospitality and Tourism, and Healthcare Services.”
The Arab Spring movements have proven that power in numbers and influence aided by the technological spread of ideas will not allow the youth cohort to be left behind in the push for development. Rather, they are demanding to be heard, and they are calling for greater capacity to be major contributors in their development goals. By funding ICT training programs that would allow youth to address the institutional weaknesses that hinder their demographic, international development organizations could find that the solution lies in shifting the goals of development towards sustainability – a sustainability that necessitates the empowerment of youth. By funding such training, youth can be heard, employed, and inform the development agendas for their countries.
We Value their Voices, and yours. What else is missing?
by Kelsey Lopez | May 16, 2014 | mobiles, Sustainable development, Unemployment, Youth
Youth Series Part II: How Technology Enables Youth Participation
See Youth Series Part I: Factors Hindering Youth Participation in Development
After discussing the various challenges youth face that inhibit their potential to be fully integrated, effective members of their societies, an analysis of some factors that enable their participation is also necessary to construct a complete picture. An International Monetary Fund (IMF) article, Youth in the Balance, elaborates on the specific challenges created by the economic crisis that spread around the world in 2008, and the subsequent responses by empowered youths – the Arab Spring. The article also notes the role of modern technology in the uprisings, stating that, “amid the turmoil and the uncertainty about their economic future, young people, more than any other group, have turned to new media for information and to communicate with their peers and beyond. Widespread access to the Internet has raised their aspirations, in part by making the young aware of the vast differences in standards of living within their countries and around the world. It has also made them more conscious of the extent of corruption and injustice and how that affects their lives” [1].
Thus, expanding access to technology has allowed for greater communication and awareness of issues, with a disproportionately higher percentage of youth using these technologies as compared to their older counterparts. As a result, there has been a surge of uprisings and political movements, spearheaded by disenfranchised youths, in response to the injustices they face, with technology being a key tool in the success of these movements in promoting change. “Whether it’s the ‘Occupy Wall Street’ movement in the United States or the mass rallies of the Arab world, young people have been jolted into action and are leading the response to diminished opportunities and unfulfilled aspirations,” a trend of empowered self-determination that has been facilitated immensely by access to technology [1]. Modern technology also aids the infectious nature of these political movements.
In cases such as the Arab Spring, youth populations have made the loudest cry for sociopolitical change, and this should come as no surprise, since the high level of unemployment most heavily affected this group in particular. As the IMF states, “youth have a huge stake in bringing about a political and economic system that heeds their aspirations, addresses their need for a decent standard of living, and offers them hope for the future,” and furthermore, “the absence of such a system is a potent recipe for conflict—especially now, with the availability of cheap means of communication such as smartphones and social media” [1].
Despite the catastrophe of youth unemployment that the global economic crisis exacerbated, technology provides a great opportunity for youths to become empowered, enabling them to participate in finding new solutions to the socioeconomic problems that plague them. As Bloom claims in Youth in the Balance, “young people will inevitably play a key role in the recovery thanks to their dynamism and willingness to relocate from labor-surplus to labor-shortage areas, and from low-productivity agriculture to higher-productivity industry and services. Their up-to-date training and education is also often a plus—although too often the education system imparts skills that are out of date or unneeded. Insofar as the expectations that education typically creates are not satisfied, youth can also power a decisive impulse to change institutions and leadership” [1]. With the heightened empowerment that access to the Internet allows, youth can take advantage of a wide variety of technological tools to achieve their own self-identified development solutions and to enable the spread of ideas.
Change is infectious, and the discontent from the ongoing Arab Spring movements is beginning to take root in other regions as well, namely Sub-Saharan Africa. In North Africa and the Middle East, “idealistic young protesters have toppled some of the most ruthless and well-resourced political strongmen on the planet,” and the conditions that led to this massive uprising are mirrored in Sub-Saharan Africa, where, “many of the underlying realities are the same. As food and fuel prices rise, inflation is driving millions of Africans below the poverty line…Radical and growing economic inequality animated much of what was at stake in the various Arab uprisings, and it will play a major role in shaping African politics for years to come” [2]. With marginalized, impoverished youths demanding new forms of government and representation, better economic policies, and greater social justice, technology plays a key role in spreading these ideas globally. Regarding the youth bulge in Africa, “their resentment is only heightened by the tools of the information age, which remind them that they have been excluded from feeding at the trough enjoyed so blatantly by the nouveau riche – a lifestyle that is showcased by the newly minted wealthy on television, Twitter, Facebook and the Web in infuriating detail. Globalization has changed the aspirations of the poor, and their expectations will follow” [2].
The feeling of inequality, or at least a perception of inequality, is a recent development challenge that the world faces, and this is a direct result of the role technology has played in highlighting the existence of inherent societal inequalities. For instance, for informed youths, “resentment of the president’s son’s Ferrari, more than envy of Europe and America’s comparative wealth, is driving young Africans into the streets to challenge their kleptocratic governments… consequently, the struggle to mitigate inequality, rather than “making poverty history” through debt relief has become the most urgent task [of development]” [2]. We live in a unique moment in development history wherein empowered youth suffrage movements were able to force the political change of the Arab Spring to a global movement. The root cause of this outrage is that, “the Arab Spring occurred at a moment when economic development had outpaced political development in much of the region; ossified political systems no longer satisfied a population yearning for modern freedoms. The explosive democratization in the Arab world, therefore, is a result of development’s success, not its failure. An authoritarian country cannot grow itself out of its fundamental underlying political contradictions. Eventually, a democratic deficit sets in” [2]. For countries that are in the midst of drastic societal change, there must be a balance of economic growth while still addressing the deeper issues that will determine the ultimate success or failure of that economic growth; these determining factors include basic freedoms, mitigating inequalities, and addressing the youths’ demands [2].
Technology has the power to harness the collective desire for youth empowerment that can bring countries out of economic crisis and into a new, more prosperous era. Central to the success of technology in aiding development goals is the transparency that it evokes, allowing cell phone and internet users to hold institutions accountable in a previously unheard of way. The challenge that remains, however, is ensuring that equal access to technology proliferates throughout all developing nations, thereby enabling citizens to communicate their ideas freely and without government interference. After establishing this holistic understanding of the ‘youth picture’ as it exists today on an institutional level, what is the challenge to development organizations that hope to combat the factors that hinder and encourage the factors that enable?
What do you think will help this movement to spread?
Sources:
[1] Bloom, D. E. (2012, March). Youth in the Balance. Retrieved from https://www.imf.org/external/pubs/ft/fandd/2012/03/bloom.htm
[2] Roos, J. (2011, July 30). Growing poverty may push Arab Spring into Africa. Retrieved from https://web.archive.org/web/20150225214042/https://roarmag.org/2011/07/growing-poverty-may-push-arab-spring-into-sub-saharan-africa/
by Kelsey Lopez | Apr 15, 2014 | Bilateral organizations, Evaluation, International aid, JICA, Results, Sustainability, Sustainable development, Uncategorized, World Bank
The lack of ex-post project evaluation at the World Bank: One has no power
The World Bank has a huge repository of 8,483 evaluation resources in its e-library database, so naturally Valuing Voices was very interested in investigating how many of those resources were ex-post evaluations of past World Bank projects. After searching the term “ex post evaluation” in the e-library, I ended up with 260 hits from those initial 8,483 resources that were a match. This looked like great news to have so many potential ex-post evaluations to analyze from such a powerhouse in international development as the World Bank. From the initial 260 hits, I expected about 50 of them to be what we consider to be ‘true’ ex-post, which is an evaluation conducted a few years after a project has been completed to assess for factors such as sustainability and long-term effectiveness of the program after donor resources had been withdrawn.
However, when I began to sift through the resources in more detail, the results were not exactly as we had anticipated. In order to determine how many of the 260 “ex-post evaluation” hits were true ex-post the process was simple, albeit time consuming. I looked through every hit, reading the abstracts provided by the World Bank and investigating individual resources in more detail if they seemed promising. While doing this, I categorized each hit by document type, keeping a tally of all the totals. The results were as follows:
|
Document Type
|
Number Encountered
|
|
Impact Evaluations
|
21
|
|
Retrospective Evaluations
|
21
|
|
Non-Evaluations (literature review, recommendations, guidelines, etc. related to evaluations)
|
53
|
|
Other (Policy reports, annual reports, sourcebooks, etc. not related to evaluations)
|
164
|
|
Ex-Post Evaluations
|
1
|
|
|
Total: 260
|
Did anything about these results surprise you? Yes, you read that right. Out of all 260 hits that came up from the search “ex-post evaluation” in the World Bank online database, an astounding grand total of one was a true ex-post evaluation of a past project. A bilateral counterpart, Japan’s JICA, has done 236 in 2009-2011 on past ODA projects, one of our rare stars in ex-post learning.
Suffice it to say, Valuing Voices was shocked by these results. There exists a clear need, based on this research, for the World Bank to contribute to the process of informing future projects by learning from past experiences, successes, mistakes, and community feedback through the valuable ex-post evaluation method. While impact and retrospective evaluations are indeed important, the nature of compiling many evaluations into one broad analysis doesn’t allow for a detailed assessment of how individual projects performed, especially when the respondents in many other multilateral ex-posts tend to remain government counterparts rather than local respondents. This type of comprehensive analysis of the long-term sustainability of completed projects can only be done by conducting ex-post evaluations for projects on a case-by-case basis.
Jindra Cekan (head of ValuingVoices) was invited to attend and speak at the World Bank’s Civil Society Organization spring meetings last week, and, armed with my findings, asked why we don’t see ex-posts at two sessions. Astrid Marroh, a senior staffer tasked with setting new strategy for the Bank, answered that longitudinal learning is, “a nut we have not yet cracked”. Varun Gauri, writing the major World Development Report 2015 on Mind and Culture at the Bank, said that not only do Bank staff need to, “change the incentives from managing projects as managers to focus on the project’s ultimate aim,” but also that the Bank, “needs to follow the private sector’s approach by ‘Dog-fooding’ our projects– living our own projects“ (where private sector producers try and eat the dog food themselves).
So what is the takeaway lesson learned form all this? Organizations like the World Bank are what set the precedent in international development, yet even this influential international organization fails to conduct regular ex-post evaluations. Despite having plentiful literature and recommendations on how to conduct ex-post evaluations and why they are important to the development process, it is clear that ex-post is not happening at the World Bank. Now is time for the organization to change the status quo and start valuing the voices of their project participants by conducting rigorous ex-post evaluations of their projects including feedback from the community level, in order to finally address this deficiency and establish a cycle of feedback loops and informed decision making that will benefit all involved – and make ‘development aid’ obsolete.
