by Jindra Cekan | Jul 15, 2021 | Accountability, Aid effectiveness, climate change, Covid, Financing gap, Impact Guild, International aid, Monitoring and Evaluation, post project evaluation, Private sector, SDGs, Sustainability, Sustainable development, Uncategorized
Sustainable Development Goals (SDGs), Funding and Accountability for sustainable projects?
What are Sustainable Development Goals? ” the United Nations adopted the new post-2015 development agenda. The new proposals – to be achieved by 2030- set 17 new ‘sustainable’ development goals (SDGs) and 169 targets. Some, like Oxfam, see the SDGs as a country budgeting and prioritization as well as an international fundraising tool. They cite that “government revenue currently funds 77% of spending…aligned with government priorities, balanced between investment and recurrent and easy to implement than donor-funded spending…” National investments are vital, but how much has the world used the SDGs to target investments and foster sustainable results?
Using results data such as that of the sectoral SDGs, countries can also ensure accountability for the policies implemented to reduce global and local inequities, but we must learn from the data. Over halfway to the goal, data is being collected, and while there is robust monitoring by countries who have built their M&E systems, other countries are faltering. “A recent report by Paris21 found even highly developed countries are still not able to report more than 40-50% of the SDG indicators” and “only 44% of SDG indicators have sufficient data for proper global and regional monitoring”. Further, there is very little evaluation or transparent accountability. Some of the data illuminate vitally need-to-know-for-better-programming. SDG data shows good news that Western and Asian countries have done better than most of the world 2015-19… but there is a lot of missing data while other data shows staggering inequities such as these:
- In Vietnam, a child born into the majority Kinh, or Viet, ethnic group is three and a half times less likely to die in his or her first five years than a child from other Vietnamese ethnic groups.
- In the United States, a black woman is four times more likely to die in childbirth than a white woman.
So are we using the SDG data to better target funding and improve design? This is the kind of evaluative learning (or at least sharing by those that are doing it :)) that is missing. As my colleague and friend Sanjeev Sridharan writes on Rethinking Evaluation, “As a field we need to more clearly understand evaluation’s role in addressing inequities and promoting inclusion” including “Promoting a Culture of Learning for Evaluation – these include focus on utilization and integration of evaluation into policy and programs.” How well learning is integrating is unknown.
As a big picture update on the progress of the Sustainable Development Goals (SDGs) in 2021, with only nine years left to the goal: It’s not looking good. The scorecards show COVID-19 has slowed down or wiped out many achievements, with 100 million people pushed into extreme poverty, according to the IMF. Pre-Covid, our blog on sectoral SDG statistics on health, poverty, hunger, and climate, was already showing very mixed results and a lack of mutual accountability.
The private sector is ever-being pushed to fund more of such development costs, only marginally successfully, as public sector expenditures are squeezed. Yet the G20 estimates that $2.5 TRILLION is needed every year to meet the SDG goals. As we have seen at Impact Guild, the push to incentivize private commitments is faltering. “To ensure its sustainability, the private sector has specific interests in securing long-term production along commodity supply chains, while reducing their environmental and social impacts and mitigating risks… The long-term economic impacts of funding projects that support the sustainability agenda are, thus, clearly understood. However, additional capital needs to flow into areas that address the risks appropriately. For example, much remains to be done to factor climate change as a risk variable into emerging markets that face the largest financing gap in achieving the SDGs.” Further, if decreased funding trends continue, by 2030, at minimum 400 million people will still live on less than $1.25 a day; around 650 million people will be undernourished, and nearly 1 billion people will be without energy access. So we’re not meeting the SDGs, they’re being derailed by COVID in places, and we aren’t beginning to cost out the need to address climate change and its effects on global development…. so now what?
From: https://www.g20-insights.org/policy_briefs/incentivizing-the-private-sector-to-support-the-united-nations-sustainable-development-goals/
To ensure that giving everyone a fair chance in life is more than just a slogan; accountability is crucial. This should include a commitment from world leaders to report on progress on “leaving no one behind” in the SDG follow-up and review framework established for the post-2015 agenda and for the private sector to loudly track their investments across the SDGs. For as The Center for American Progress wrote, money and results are key: We must “measure success in terms of outcomes for people, rather than in inputs—such as the amount of money spent on a project—as well as in terms of national or global outcomes” and that “policymakers at the global level and in each country should task a support team of researchers with undertaking an analysis of each commitment.”
A further concern. While we seem to measure the statistics periodically and see funding allocated to SDG priorities, but there are few causal links drawn between intensity in investment in any SDG goal and sustained results. To what degree are the donations/ investments into the SDGs linked to improvements? Without measuring causality or attribution, it could be a case of “A rising tide lifts all boats” as economies improve or, as Covid-related economic decline wiped out 20 years of development gains as Bill Gates noted last year. We need proof that trillions of dollars of international “Sustainable development” programs have any sustained impact beyond the years of intervention.
We must do more evaluation and learn from SDG data for better targeting of investments and do ex-post sustainability evaluations to see what was most sustained, impactful, and relevant. Donors should raise more funds to meet needs and consider only funding what could be sustained locally. Given the still uncounted demands on global development funding, we can no longer hope or wait for global mobilization of trillions given multiple crises pushing more of the world into crisis. Let’s focus now.
by valuingvoicesjin | Sep 24, 2019 | Uncategorized
Accountability: Are we responsible for meeting the SDGs? Yes, personally and by programming for more sustainability
In a 2014 article about mutual accountability for the SDGs, Dr. Paul Zeitz states that “Sustainable development is the most urgent challenge facing humanity. The fundamental question is how the world economy can continue to develop in a way that is socially inclusive, advances human rights, and ensures environmental sustainability.” Today, the UN Secretary General’s report on SDG Progress on progress to meeting them is unveiled. As many of us work in global development, we need to consider our accountability for its findings.
First, Dr. Zeitz points to key aspects to such accountability, of which three are most relevant to sustainability:
- Universal, Voluntary and Commitment-Based Approach
For the SDG agenda to be successful, “Shared and joint commitments by partners from governments, civil society and the private sector can inspire faster and bolder action, can garner enhanced citizen and media attention; and can contribute to the mobilization of resources from internal and external sources.”
Our industry has pushed for greater investments for decades.
- Broad-Based Youth and Citizen Engagement
Local youth- and citizen-driven monitoring and accountability mechanisms are essential for improving budget transparency and service delivery outcomes. If citizens are enabled to pay attention, respond and engage, and then take responsibility and action, then everyone can be empowered to foster an enabling environment for “mutual accountability” and measurable results.”
While most of our work does M&E and listen to ciitzens, even foster voice through civic engagement and feedback loops, we are far from done.
- Call for a Multi-Stakeholder SDG Monitoring and Accountability Mechanism
“… it is more challenging and more complex to ensure ‘mutual accountability’ for results. Given the advances in human cooperation and technology, we know that the SDG era can usher in and foster a new culture of ‘mutual accountability.’
It is here where we fail quite badly in global aid. We rarely talk about our reciprocal accountability with our participants and partners, with the countries themselves. Too often we push money, extract data, claim success and leave abruptly. These SDGs push us to think about “global accountability” and how our actions at work and home need to change to (un)affect others. In a piece on mutual accountability, “Accountability for Development Cooperation under the 2030 Agenda” by Timo Mahn Jones explores “global accountability”, based on “mutual accountability, by which two partners agree to be held responsible for the commitments they voluntarily made.”
- He warns of the danger that “development cooperation stakeholders do not follow through with their commitments, and are not held accountable.
- He suggests that existing donors need to honor ODA/aid targets and new partners, especially the private sector are vital for the implementation of the 2030 Agenda, “to move from billions to trillions” in funding.
- Equally in need of revision is the roles of ‘donor’ and ‘recipient’ as we have entered a new world of “sharing of risk and ‘mutual pain’ where all of us are affected if we do not reach the SDG targets – although unequally.
How do those of us in our industry foster such accountability? We can say, rightly that we have played a role in sectors like health, agriculture, natural resource management through many projects over the last century or more. Absolutely but we know, our projects are piecemeal and often scattered, more short than long in implementation (typically 5 years) and as we rarely return after clcose-out, we do not know how long will results will be sustained. Yet we have tried to do good and there are many public and private players. Is it adding up? .
The Sept 2019 UN report shows promising if mixed results.
Immunizations are increasing, thereby saving millions of lives (SDG3).
There is also good news from investments in renewable energy growth.
However, while those living below minimum living wage is falling on every continent, still, “one child in five lives in extreme poverty” (SDG1).
Hunger, unfortunately, is rising (SDG2), partly due to a 150% increase in “direct economic losses from disasters… over the past 20 years, with losses disproportionately borne by vulnerable developing countries” and 68.5 million people have been forcibly displaced, sometimes from wars donor governments supply arms to or refugees are forced to remain in other developing countries, the UN report tells us, with shrinking refugee funds.
Climate is the most worrisome of the UN reports. Reversing CO2 emissions and fostering the sustainability of climate adaptation and mitigation projects is imperative given today’s UNEP report that includes this graph. Collective accountability is key to reining in CO2, for our emissions are leading to unprecedented ice melt, sea-level rise, and high pollution, not globally sustainable.
While we in ‘development’ could say we affect the earlier SDGs, climate is a global problem, one that each one of us affects with our consumptive actions… or aren’t all of them? Don’t we affect hunger through our food waste and food purchases from afar? Don’t we affect child survival and immunization through our advocacy for aid and even private donations to other health and food security charities?
Some of us disproportionally affect emissions by our wealth, population, or both. The Global Carbon Atlas from 2017 shows how wealthier and more populous countries emit far more than the poor ones. As CO2 rises, climate is affected, storms are more severe, yet the poorest countries have the fewest means to prepare or respond. In their World Disasters Report, The International Federation of the Red Cross found that “Between 1991 and 2010, the impact of recorded disaster events in poor countries resulted in over $840 billion of financial losses. Yet, over the same period, only 0.4% of the $3.3 trillion spent on aid was dedicated to prevention or risk reduction“. Our industry does some Disaster Risk Reduction (DRR) but why have we not advocated to prevent more suffering?
Thinking about “collective accountability” makes this graphic from the UN SDG report uncomfortable, as the burden of climate change falls disproportionately on the poor through economic losses from disasters.
The UN Secretary General states: “The world will soon enter a decade that will be decisive for both current and future generations and for all life on this planet. It is the world’s responsibility and within its power to make it a decade of action and delivery for sustainable development.”
This will require both a clear accountability to them – not just us and ‘our’ projects. This will require us seeing such projects as our continued responsibility to sustain, namely, to design them collaboratively enough, led by local partners ranging from governments and private sector to communities, with the means to sustain what they value. As we return to evaluate sustainability less than 1% of the time, learning opportunities have been scarce to improve current and future projects.
USAID has talked about long-term transparent and accountable investments in “local solutions” partners. Has it worked? Not yet. While President Obama and former United States Agency for International Development (USAID) Administrator Raj Shah promised up to 30% of all contracts would go to ‘local solutions’ that “promote sustainable development through high-impact partnerships and local solutions”, little of that was met, given bureaucracy and the need for fast success, rather than investing more in long-term capacity development of partners. While there seem to be good examples such as Haiti, Afghanistan is a poorer example. While most international non-profits implement projects through local sub-contractors, certainly building their capacity to manage and account for foreign taxpayer dollars spent, like this MSI in Lebanon example, is important. But if we extend the measure of ‘success’ beyond our project implementation, then policies and programming needs to change to sustain capacity and implementation post-exit (INTRAC report). Too often we still exit when funds are spent. USAID’s new Journey to Self-Reliance does promise to listen, to “ support partners to become self-reliant and capable of leading their own development journeys.“
It requires listening to those whom our aid, aids. Time to Listen talks movingly of a desire for collaboration during design, implementation, monitoring and evaluation. It requires we take the time to listen, have an openness to learn from our partners and participants about what they could – and could not- sustain and why. We need to do so before, during and well-after projects close, learn quickly and do better and be collectively accountable for longer. It requires seeing such people as real experts, not abstractions. How? Listening to those living with hunger and climate change is vital. While we read about Amazonian fires threatening over 2 million acres of rainforest, contextualizing statistics with stories that illustrate that those whose Development Goals donors are ‘sustaining’ know best what works there. In this case, understanding the range of why the Amazon rainforest reserves are endangered is important for design, so that better approaches to achieving SDGs become a driving force to change all our lives. We need to be accountable to them.
Let me know your thoughts on bringing the SDGs to our work and lives…
by Kelsey Lopez | Mar 2, 2015 | Accountability, Beneficiaries, Evaluation, Evidence-based policy, ex-post evaluation, Feedback loops, impact evaluation, Local Participants, Participants, Participation, post-project evaluation, Results, self-sustainability, Sustainability, Sustainable development, Transparency, Uncategorized, Valuing Voices, World Bank
Pick a term, any term…but stick to it!
Valuing Voices is interested in identifying learning leaders in international development that are using participatory post-project evaluation methods to learn about the sustainability of their development projects. These organizations not only believe they need to see the sustained impact of their projects by learning from what has worked and what hasn’t in the past, but also that participants are the most knowledgeable about such impacts. So how do they define sustainability? This is determined by asking questions such as the following: were project goals self-sustained by the ‘beneficiary’ communities that implemented these projects? By our VV definition, self-sustainability can only be determined by going back to the project site, 2-5 years after project closeout, to speak directly with the community about the long-term intended/unintended impacts.
Naturally, we turned to the World Bank (WB) – the world’s prominent development institution – to see if this powerhouse of development, both in terms of annual monetary investment and global breadth of influence, has effectively involved local communities in the evaluation of sustainable (or unsustainable) outcomes. Specifically, my research was focused on identifying the degree to which participatory post-project evaluation was happening at the WB.
A fantastic blog* regarding participatory evaluation methods at the WB emphasizes the WB’s stated desire to improve development effectiveness by “ensuring all views are considered in participatory evaluation,” particularly through its community driven development projects. As Heider points out,
“The World Bank Group wants to improve its development effectiveness by, among others things, engaging citizens throughout the operational project cycle. It has set itself an ambitious target: 100% citizen engagement in projects that have clearly identifiable beneficiaries.”
Wow! Though these methods are clearly well intentioned, there seems to be a flaw in the terminology. The IEG says, “[Community driven development projects] are based on beneficiary participation from design through implementation, which make them a good example of citizen-centered assessment techniques in evaluation,” …however, this fails to recognize the importance of planning for community-driven post-project sustainability evaluations, to be conducted by the organization in order to collect valuable data concerning the long-term intended/unintended impacts of development work.
With the intention of identifying evidence of the above-mentioned mode of evaluation at the WB, my research process involved analyzing the resources provided by the WB’s Independent Evaluation Group (IEG) database of evaluations. As the accountability branch of the World Bank Group, the IEG works to gather institution-wide knowledge about the outcomes of the WBs finished projects. Its mission statement is as follows:
“The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group’s work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings.”
Another important function of the IEG database is to provide information for the public and external development organizations to access and learn from; this wealth of data and information about the World Bank’s findings is freely accessible online.
When searching for evidence of post-project learning, I was surprised to find that the taxonomy varied greatly; e.g. projects I was looking for could be found under ‘post-project’, post project’, ‘ex-post’ or ‘ex post’. What was also unclear was any specific category under which these could be found, including a definition of what exactly is required in an IEG ex post impact evaluation. According to the IEG, there are 13 major evaluation categories, which are described in more detail here. I was expecting to find an explicit category dedicated to post-project sustainability, but instead this type of evaluation was included under Project Level Evaluations (which include PPARs and ICRs [Implementation Completion Reports]), and Impact evaluations.
This made it difficult to determine a clear procedural standard for documents reporting sustainability outcomes and other important data for the entire WB.
I began my research process by simply querying a few key terms into the database. In the first step of my research, which will be elaborated upon in Part I in this blog series, I attempted to identify evidence of ex post sustainability evaluation at the IEG by searching for the term “post-project” in the database, which yielded 73 results when using a hyphen and 953 results without using a hyphen. I found it interesting the inconsistency in the number of results depending on the use of a hyphen, but in order to narrow the search parameters to conduct a manageable content analysis of the documents, I chose to breakdown these 73 results by document type to determine if there are any examples of primary fieldwork research. In these documents, the term “post-project” was not used in the title of the documents or referenced in the executive summary as the specific aim of the evaluation, but rather used to loosely define the ex post time frame. Figure 1 illustrates the breakdown of document types found in the sample of 73 documents that came up when I searched for the key term “post-project”:
Figure 1: Breakdown by Document Type out of Total 73 Results when searching post-project
As the chart suggests, many of the documents (56% – which accounts for all of the pie chart slices except Project Level Evaluations) were purely desk studies – evaluating WB programs and the overall effectiveness of organization policies. These desk studies draw data from existing reports, such as those published at project closeout, without supplementing past data with new fieldwork research.
Out of the 9 categories, the only document type that showed evidence of any follow up evaluations were the Project Performance Assessment Reports (PPARs), defined by the IEG as documents that are…
“…based on a review of the Implementation Completion Report (a self-evaluation by the responsible Bank department) and fieldwork conducted by OED [Operations Evaluation Department]. To prepare PPARs, OED staff examines project files and other documents, interview operational staff, and in most cases visit the borrowing country for onsite discussions with project staff and beneficiaries. The PPAR thereby seeks to validate and augment the information provided in the ICR, as well as examine issues of special interest to broader OED studies.”
Bingo. This is what we’re looking for. The PPARs accounted for 32 out of the 73 results, or a total of 44%. As I examined the methodology used to conduct PPARs, I found that in the 32 cases that came up when I searched for “post-project”, after Bank funds were “fully dispersed to a project” and resources were withdrawn, the IEG sent a post-project mission back into the field to collaborate on new M&E with local stakeholders and beneficiaries. The IEG gathered new data through the use of field surveys or interviews to determine project effectiveness.
Based on these findings, I conducted a supplementary search of the term “ex post”, which yielded 672 results. From this search, 11 documents were categorized by the IEG as “Impact Evaluations”, of which 3 showed evidence of talking with participants to evaluate for sustainability outcomes. In follow-up blogs in this series I will elaborate upon the significance of these additional findings and go into greater detail regarding the quality of the data in these 32 PPARs, but here are a few key takeaways from this preliminary research:
-
Taxonomy and definition of ex-post is missing. After committing approximately 15-20 hours of research time to this content analysis, it is clear that navigating the IEG database to search for methodology standards to evaluate for sustainability is a more complicated process than it should be for such a prominent learning institution. The vague taxonomy used to categorize post-project/ex-post evaluation by the WB limits the functionality of this resource as a public archive dedicated to informing the sustainability of development projects the World Bank has funded.
-
Despite affirmative evidence of participatory community involvement in the post-project evaluation of WB projects, not all PPARs in the IEG database demonstrated a uniform level of ‘beneficiary’ participation. In most cases, it was unclear how many community members impacted by the project were really involved in the ex-post process, which made it difficult to determine even a general range of the number of participants involved in post-project activity at the WB.
-
Although PPARs report findings based, in part, on post-project missions (as indicated in the preface of the reports), the specific methods/structure of the processes were not described, and oftentimes the participants were not explicitly referenced in the reports. (More detailed analysis on this topic to come in Blog Series Part 2!)
-
These surprisingly inconsistent approaches make it difficult to compare results across this evaluation type, as there is no precise status quo.
Finally, the World Bank, which has funded 12,000 projects since its inception, should have far more than 73 post-project/ ex-post evaluations…but maybe I’m just quibbling with terms.
Stay tuned for PART II of this series, coming soon!
by Kelsey Lopez | Apr 15, 2014 | Bilateral organizations, Evaluation, International aid, JICA, Results, Sustainability, Sustainable development, Uncategorized, World Bank
The lack of ex-post project evaluation at the World Bank: One has no power
The World Bank has a huge repository of 8,483 evaluation resources in its e-library database, so naturally Valuing Voices was very interested in investigating how many of those resources were ex-post evaluations of past World Bank projects. After searching the term “ex post evaluation” in the e-library, I ended up with 260 hits from those initial 8,483 resources that were a match. This looked like great news to have so many potential ex-post evaluations to analyze from such a powerhouse in international development as the World Bank. From the initial 260 hits, I expected about 50 of them to be what we consider to be ‘true’ ex-post, which is an evaluation conducted a few years after a project has been completed to assess for factors such as sustainability and long-term effectiveness of the program after donor resources had been withdrawn.
However, when I began to sift through the resources in more detail, the results were not exactly as we had anticipated. In order to determine how many of the 260 “ex-post evaluation” hits were true ex-post the process was simple, albeit time consuming. I looked through every hit, reading the abstracts provided by the World Bank and investigating individual resources in more detail if they seemed promising. While doing this, I categorized each hit by document type, keeping a tally of all the totals. The results were as follows:
|
Document Type
|
Number Encountered
|
|
Impact Evaluations
|
21
|
|
Retrospective Evaluations
|
21
|
|
Non-Evaluations (literature review, recommendations, guidelines, etc. related to evaluations)
|
53
|
|
Other (Policy reports, annual reports, sourcebooks, etc. not related to evaluations)
|
164
|
|
Ex-Post Evaluations
|
1
|
|
|
Total: 260
|
Did anything about these results surprise you? Yes, you read that right. Out of all 260 hits that came up from the search “ex-post evaluation” in the World Bank online database, an astounding grand total of one was a true ex-post evaluation of a past project. A bilateral counterpart, Japan’s JICA, has done 236 in 2009-2011 on past ODA projects, one of our rare stars in ex-post learning.
Suffice it to say, Valuing Voices was shocked by these results. There exists a clear need, based on this research, for the World Bank to contribute to the process of informing future projects by learning from past experiences, successes, mistakes, and community feedback through the valuable ex-post evaluation method. While impact and retrospective evaluations are indeed important, the nature of compiling many evaluations into one broad analysis doesn’t allow for a detailed assessment of how individual projects performed, especially when the respondents in many other multilateral ex-posts tend to remain government counterparts rather than local respondents. This type of comprehensive analysis of the long-term sustainability of completed projects can only be done by conducting ex-post evaluations for projects on a case-by-case basis.
Jindra Cekan (head of ValuingVoices) was invited to attend and speak at the World Bank’s Civil Society Organization spring meetings last week, and, armed with my findings, asked why we don’t see ex-posts at two sessions. Astrid Marroh, a senior staffer tasked with setting new strategy for the Bank, answered that longitudinal learning is, “a nut we have not yet cracked”. Varun Gauri, writing the major World Development Report 2015 on Mind and Culture at the Bank, said that not only do Bank staff need to, “change the incentives from managing projects as managers to focus on the project’s ultimate aim,” but also that the Bank, “needs to follow the private sector’s approach by ‘Dog-fooding’ our projects– living our own projects“ (where private sector producers try and eat the dog food themselves).
So what is the takeaway lesson learned form all this? Organizations like the World Bank are what set the precedent in international development, yet even this influential international organization fails to conduct regular ex-post evaluations. Despite having plentiful literature and recommendations on how to conduct ex-post evaluations and why they are important to the development process, it is clear that ex-post is not happening at the World Bank. Now is time for the organization to change the status quo and start valuing the voices of their project participants by conducting rigorous ex-post evaluations of their projects including feedback from the community level, in order to finally address this deficiency and establish a cycle of feedback loops and informed decision making that will benefit all involved – and make ‘development aid’ obsolete.
by Siobhan Green | Apr 7, 2014 | Participation, Sustainability, Sustainable development, Uncategorized
What to consider for sustainability….
Recently I read an article on “how to make a computer lab sustainable”. I read it with interest because I find the question about sustainability in international development fascinating. What do we mean by sustainability? Every project has a requirement to be sustainable, but the definition seems to vary based on the project or the donor. Does it mean “continue to exist beyond the period of performance?” “Positive impacts continue beyond the period of performance?”, “Not just continue to exist but grow and scale?”
Do we mean sustainability of operations? Of financial independence? Impact? Involvement of local populations? All of the above?
In the article I mentioned, sustainability really meant the hardware and software would continue beyond the funding cycle and that they were looking at fee for service opportunities/integration into an existing organization. But they didn’t mention any details about the business model –would the fee for service cover the increased costs for internet, hardware/software support and purchases, salaries of staff, electricity? Would they need to find additional sponsorship from donors or partner organizations? Do they have a management team which is capable of managing the budget, of ensuring high quality of services to customers, of finding new sponsors? What about growth and changes in technology infrastructure?
Now, the project may well have included these aspects in their design, but the article didn’t mention any of these things. And for me, sustainability starts with thinking about treating the seed funding you get as an initial investment, which should lead to a self-sustaining model that won’t require more investment (at least not for the same types of activities). If the model involves transitioning to local staff or partner organization, that needs to be both explicit and planned from the start, with explicit assumptions and expectations on what sustainability means.
Because otherwise sustainability is kind of a meaningless term.
Who should decide what sustainability is unless we Value all Voices to discuss and decide? What are your thoughts?
by Kelsey Lopez | Mar 7, 2014 | Bilateral organizations, Donors, Evaluation, International aid, International non-profits, JICA, Local Participants, OECD, Participation, Results, Sustainability, Uncategorized
What are the prospects for self-sustained impact at the Macro level? (Part 1)
After studying various impact evaluations, it became clear that different development projects also have different scales, with multilaterally-funded macro-level being focused on higher scale objectives from the municipal up to the national level while micro-level are geared towards improving socioeconomic aspects of communities. Our research revealed that there is a recurring error among projects that makes them ineffective: a lack of community involvement in all stages of the project, or more specifically, their lack of involvement in both the design and the evaluation process. This results in a lack of opinions from the local level regarding project self-sustainable goals, and prospects for ultimate success in self-sustainability. We have seen both macro projects, implemented by multilaterals, and micro projects, implemented by international non-profits, that fail to include local participation in their methodology, but one trend we have noticed is that projects on the macro level, have a tendency to plan, implement, and complete projects with little input from local participants or consideration for the possibility of self-sustainability impacts.
A particularly egregious example of non-self-sustainability at any level is the project for OECD-funded Improving the Performances and Management of Public Lighting in Ho Chi Minh City, which was a project with no ground level/community participation, but rather took place at the municipal level between French company Citelum (supplying technical assistance and equipment) and Sapulico (Saigon Public Lighting Company, Ho Chi Minh City's public lighting authority). From the evaluation summary, we saw the project was most concerned with propagating appreciation for French expertise in the area of public lighting at the national level rather than fostering national electrical self-sustainability. With organizational sustainability as a main objective, the project itself was not considered self-sustainable for a few key reasons:
-
“Sapulico's engineers acquired the necessary [skills required to operate the system], however, their current skill level is insufficient to handle present and future technical difficulties without the assistance of Citelum.” This means that the local company in Vietnam would continue to be reliant on foreign expertise in the likely event of future technical difficulties.
-
Also, jarringly, “there are also concerns even about the project's current viability as Sapulico's budget doesn't allow for the purchase of equipment from abroad, therefore the company will be unable to replace equipment that has reached the end of its lifecycle.”
This highlights an obvious overlooking of future project sustainability due to the local budget conditions and insufficient training. Had the project been more concerned with guaranteeing even national self-sustainability rather than the financial sustainability of the foreign implementing organization, the project overall could have been more successful far into the future….
A more promising example of a macro level project aiming for more localized impact, yet still failing to consider self-sustainability, is the JICA Agricultural Development Project in Kambia District. While we commend JICA as the most active multilateral to do ex-post evaluation, we again see the tendency of macro level projects to focus on systemic changes at the national or at least regional level which often self-sustainability buy-in, whereas micro level projects tend to focus on communities or at most sub-regions with the objective of fostering longer-term sustainability. This project, on the other hand, is a hybrid of macro- and micro- as it was to intervene at a district level and it did try to create agricultural extension buy-in. The key takeaways from JICAs post-project evaluation:
-
This project was concerned with improving agricultural productivity at select sub-district pilot sites, and then a further extension of the project throughout the entire Kambia District. The evaluation focuses heavily on the inability of the project to complete this objective of extending the project geographically because there was insufficient manpower, technical support, and funding to complete project dissemination, which suggests that sustained impact was not fully considered when planning implementation. It also suggests it did not expand because lessons seem not to have been learned from the implementation, and funding was not longer available.
-
The key point here is that the project did not secure adequate resources or funding for the true completion of its objective, which was to extend the successful outcomes that they observed at the pilot sites further across the region. This resulted in an evaluation analysis of sustainability that was only “fair”.
-
This project also fails to mention any local participation in the design, planning, implementation, or long-term sustainability phases, so despite seeing successful outcomes in increased crop production at the pilot locations, the evaluation was ultimately negative.
-
It is clear from the evaluation that the project was being executed from the side of the implementing agency rather than involving local participation beyond the extensionists, and no input was asked from community members about resulting agricultural self-sustainability.
-
The crux is that had the project been more focused on local self-sustainability beyond the regional extension system, its own organizational deficiencies (lack of manpower, technical support, and funding mentioned above) might not have hindered the ability to expand the project further.
Compare these findings to those we have analyzed in other blogs which illustrated the far more successful project and evaluation results that micro level non-profit development agencies like Mercy Corps, Plan International, and Partners for Democratic Change fostered when they made community participation the key to their development process. However, JICAs Kambia project and the Vietnamese Public Lighting program are not alone in their shortcomings, because the consistent lack of local community involvement in community-focused projects is a theme that we continue to see with macro level and micro level development schemes – a pattern that Valuing Voices is determined to change.
Overall, what we have discerned from this analysis is that projects conducted by macro level multilateral actors have a lower chance of ensuring self-sustainability, in part because they usually aren’t linked directly to the community in regional or national-scale projects. By contrast, micro level projects by international non-profits tend to have a higher chance of fostering greater self-sustainability because they are linked more closely to the end users of development, thereby benefiting both the organization itself as well as the local participants from more inclusionary practices.
We can clearly see that at least a few organizations, both at the macro and micro level, are asking the right questions by conducting these post-project evaluations. However, a systemic problem persists in that virtually no one is designing for self-sustainability. But without it, how successful are we really?
