by Kelsey Lopez | Mar 7, 2014 | Bilateral organizations, Donors, Evaluation, International aid, International non-profits, JICA, Local Participants, OECD, Participation, Results, Sustainability, Uncategorized
What are the prospects for self-sustained impact at the Macro level? (Part 1)
After studying various impact evaluations, it became clear that different development projects also have different scales, with multilaterally-funded macro-level being focused on higher scale objectives from the municipal up to the national level while micro-level are geared towards improving socioeconomic aspects of communities. Our research revealed that there is a recurring error among projects that makes them ineffective: a lack of community involvement in all stages of the project, or more specifically, their lack of involvement in both the design and the evaluation process. This results in a lack of opinions from the local level regarding project self-sustainable goals, and prospects for ultimate success in self-sustainability. We have seen both macro projects, implemented by multilaterals, and micro projects, implemented by international non-profits, that fail to include local participation in their methodology, but one trend we have noticed is that projects on the macro level, have a tendency to plan, implement, and complete projects with little input from local participants or consideration for the possibility of self-sustainability impacts.
A particularly egregious example of non-self-sustainability at any level is the project for OECD-funded Improving the Performances and Management of Public Lighting in Ho Chi Minh City, which was a project with no ground level/community participation, but rather took place at the municipal level between French company Citelum (supplying technical assistance and equipment) and Sapulico (Saigon Public Lighting Company, Ho Chi Minh City's public lighting authority). From the evaluation summary, we saw the project was most concerned with propagating appreciation for French expertise in the area of public lighting at the national level rather than fostering national electrical self-sustainability. With organizational sustainability as a main objective, the project itself was not considered self-sustainable for a few key reasons:
-
“Sapulico's engineers acquired the necessary [skills required to operate the system], however, their current skill level is insufficient to handle present and future technical difficulties without the assistance of Citelum.” This means that the local company in Vietnam would continue to be reliant on foreign expertise in the likely event of future technical difficulties.
-
Also, jarringly, “there are also concerns even about the project's current viability as Sapulico's budget doesn't allow for the purchase of equipment from abroad, therefore the company will be unable to replace equipment that has reached the end of its lifecycle.”
This highlights an obvious overlooking of future project sustainability due to the local budget conditions and insufficient training. Had the project been more concerned with guaranteeing even national self-sustainability rather than the financial sustainability of the foreign implementing organization, the project overall could have been more successful far into the future….
A more promising example of a macro level project aiming for more localized impact, yet still failing to consider self-sustainability, is the JICA Agricultural Development Project in Kambia District. While we commend JICA as the most active multilateral to do ex-post evaluation, we again see the tendency of macro level projects to focus on systemic changes at the national or at least regional level which often self-sustainability buy-in, whereas micro level projects tend to focus on communities or at most sub-regions with the objective of fostering longer-term sustainability. This project, on the other hand, is a hybrid of macro- and micro- as it was to intervene at a district level and it did try to create agricultural extension buy-in. The key takeaways from JICAs post-project evaluation:
-
This project was concerned with improving agricultural productivity at select sub-district pilot sites, and then a further extension of the project throughout the entire Kambia District. The evaluation focuses heavily on the inability of the project to complete this objective of extending the project geographically because there was insufficient manpower, technical support, and funding to complete project dissemination, which suggests that sustained impact was not fully considered when planning implementation. It also suggests it did not expand because lessons seem not to have been learned from the implementation, and funding was not longer available.
-
The key point here is that the project did not secure adequate resources or funding for the true completion of its objective, which was to extend the successful outcomes that they observed at the pilot sites further across the region. This resulted in an evaluation analysis of sustainability that was only “fair”.
-
This project also fails to mention any local participation in the design, planning, implementation, or long-term sustainability phases, so despite seeing successful outcomes in increased crop production at the pilot locations, the evaluation was ultimately negative.
-
It is clear from the evaluation that the project was being executed from the side of the implementing agency rather than involving local participation beyond the extensionists, and no input was asked from community members about resulting agricultural self-sustainability.
-
The crux is that had the project been more focused on local self-sustainability beyond the regional extension system, its own organizational deficiencies (lack of manpower, technical support, and funding mentioned above) might not have hindered the ability to expand the project further.
Compare these findings to those we have analyzed in other blogs which illustrated the far more successful project and evaluation results that micro level non-profit development agencies like Mercy Corps, Plan International, and Partners for Democratic Change fostered when they made community participation the key to their development process. However, JICAs Kambia project and the Vietnamese Public Lighting program are not alone in their shortcomings, because the consistent lack of local community involvement in community-focused projects is a theme that we continue to see with macro level and micro level development schemes – a pattern that Valuing Voices is determined to change.
Overall, what we have discerned from this analysis is that projects conducted by macro level multilateral actors have a lower chance of ensuring self-sustainability, in part because they usually aren’t linked directly to the community in regional or national-scale projects. By contrast, micro level projects by international non-profits tend to have a higher chance of fostering greater self-sustainability because they are linked more closely to the end users of development, thereby benefiting both the organization itself as well as the local participants from more inclusionary practices.
We can clearly see that at least a few organizations, both at the macro and micro level, are asking the right questions by conducting these post-project evaluations. However, a systemic problem persists in that virtually no one is designing for self-sustainability. But without it, how successful are we really?
by Kelsey Lopez | Feb 27, 2014 | Evaluation, Governance, Local Participants, Participation, Results, Sustainability
Catalyst organizations are those whose focus is on implementing programs with community level involvement during projects and local feedback loops to inform post-project evaluations for impact self-sustainability. An excellent example of this is Partners for Democratic Change, whose stated mission is, “to build sustainable capacity to advance civil society and a culture of change and conflict management worldwide,” focusing on initiating democratic practices through an approach called Sustainable Impact Investing. The goal of this approach is to foster the capacity of in-country organizations to “deliver systematic change,” with a focus on development that, “is bottom-up, locally-led rather than foreign-led, based on the belief that change comes from sustainable efforts led by local people, organizations and institutions invested in their own long-term future.”
To implement this progressive and participatory vision for sustainable development, Partners’ founded 22 Centers for Change and Conflict Management between 1989-2011, initially in the regions of Eastern and Central Europe. They later expanded to other regions struggling with democratic sociopolitical change. Partners’ conducted its own ex-post evaluation that averaged the results of 55 case studies that led to positive significant outcomes. The takeaway resulted in three main sustainability lessons:
-
“The importance of investing in local partners and building their capacity to promote democratic change;
-
the most pressing development challenges facing the world need to be addressed in a participatory manner with the input and shared commitment of government, businesses and civil society, which requires local leaders with sophisticated skills in change and conflict management;
-
and finally, the work of social entrepreneurs to make a difference in their own countries is strengthened and legitimated by technical and relational support from an international network of like-minded professionals facing similar challenges.”
With these objectives in mind, the greatest positive outcome that was observed occurred in almost 90% of the 55 stories. This outcome was that development and participation of civil society is most commonly achieved through, “education, training, mentoring, coaching, partnerships and coalition building, organizational development and capacity building, and creating and enabling environment that supports civil society development, such as passing NGO laws.” Further, in 80% of cases, there was advancement of good governance by influencing the participation of civil society working with government on the issues listed above, specifically free and fair elections, human rights protection, etc. Another 50% of the cases increased access to justice and managing and resolving disputes/conflicts, thereby strengthening civil society, and about 40% of the stories focused on promoting inclusive societies, improving majority-minority relations, and increasing leadership capacity for women and youth as agents for social change. The overall result of Partners’ efforts resulted in substantial impacts. Since 1991, the Centers have trained around 15,000 mediators and worked directly with more than 300,000 participants, benefitting an estimated total of 17.5 million people – and these are even considered to be conservative estimates. 22 total Centers had been established, and 18 still exist today with a success rate of 82%.
Yet the Centers still faced challenges, most notably in, “institutionalizing the processes they used to achieve results so that impact can be maximized and sustainable.” While the Centers effectively managed to implement collaborative and participatory methods to attain these successful outcomes, without the ability to institutionalize these processes in local communities and government institutions, the likelihood of sustainability is threatened.
Herein lies the importance of valuing local voices and participation, as it is clear that successful development initiatives depend on working with the community rather than on behalf of the community. Collaborative efforts between local participants and the international organizations that aim to enhance socioeconomic development in their communities results in both farther reaching and more sustainable outcomes than projects that ignore local feedback. Partners also does a great job bridging the objectives of building organizational capacity to sustain programming while also ValuingVoices of participants regarding how that capacity will be beneficial to them. There is an obvious need throughout the development community to follow the good examples made by Partners for Democratic Change in order to promote greater levels of participation on the path to sustainable development.
by Kelsey Lopez | Feb 12, 2014 | Evaluation, Food security, International non-profits, Literacy, Local Participants, Participation, Results, Sustainability, Uncategorized
What can we learn from Ex-Post Evaluations?
In trying to learn more about sustainable development solutions, the first place to look for information is in ex-post evaluations, also commonly called post-project evaluations, which are conducted by either development organizations themselves or by independent external evaluators. Unlike final project evaluations, which are completed at the time of a project’s conclusion to assess whether or not it has achieved its intended goals, an ex-post evaluation is conducted in the years after a project’s official end date – maybe one, three, or five years after the fact. An ex-post evaluation is a highly valuable tool for determining not just how successful a development project may have been after resources and international funding were withdrawn, but rather the long-term sustainability of the outcomes for the community members who were being ‘developed’.
With the seemingly obvious necessity for ex-post evaluations to gaining a better understanding of both positive and negative development practice, I was surprised by how hard it was to actually find any. Some organizations are diligent about conducting post-project evaluations and documenting the results for future reference, namely the development assistance organization Japan International Cooperation Agency (JICA), which has an extensive reference database to search its ex-post evaluations. However, this is certainly not the norm (yet), or if organizations are conducting ex-post evaluations they are not making the information widely available to the public. My research process included search terms such as, “ex-post evaluations by international development organizations”, “post-project evaluations”, and “impact evaluations.” In using these generic search terms, I was only moderately successful in finding helpful evaluations for my reserch, which suggests the need for more readily accessible information to the public about development outcomes.
We also found that some organizations had completed these evaluations, but they were at times too vague to obtain much useful information from. Out of about 10-15 evaluations that we found so far, there were only around 7 that were clear and organized enough to include in my table of summaries. (My search was limited to projects that were conducted predominantly at the community level, rather than at the municipal or state level.) The variable quality of these evaluations has a negative impact on their usefulness – if an ex-post evaluation is in an unsearchable format or doesn’t follow a fairly standardized organization, how will it be able to inform future projects efficiently? Additionally, it would be much easier for project coordinators to learn from past projects, and even other organizations, if there existed a more accessible and methodical database to make searching for ex-post evaluations simple. Despite these challenges, I have included five different evaluations from my preliminary research with which I was able to compare results for a better understanding of how to achieve sustainable project outcomes. The framework used for analyzing these evaluations considered:
- The sector of the development project (i.e. food security, poverty reduction, agricultrual development);
- the implementing organization and the evaluating organization (if it was different);
- the dates and gap between the project and the ex-post evaluation;
- the project objectives;
- specific ex-post evaluation methods;
- the positive/sustainable outcomes;
- the negative/unsustainable outcomes;
- the transfer to authorities;
- the amount of money invested overall;
- and the level of local participation.
The five evaluations analyzed include:
For a full summary of these evaluations, please see the Ex-Post Evaluations Summary Table. Here, are brief synopses of the most pertinent information for the above framework of analysis, and the table provides a better context for our conclusions.
Here are the key findings from the various ex-post evaluations that we found to be most significant:
- Over 18 million USD were spent on the five combined projects, but most projects did not explicitly enumerate how many people/households were impacted by the individual projects. An exception to this is the project in Mauritius, which reported reaching around 3,500 people. Without understanding the scale of the program, it is difficult to compare projects directly to one another.
- Mercy Corps’ MILK Project in Niger was inclusive and participatory in its ex-post evaluation process, which resulted in hard data that can easily be analyzed, compared, and learned from in future projects. In addition, this evaluation utilized a unique pictoral tool developed specifically to include all project participants in the feedback loop, despite widespread illiteracy, so that every individual had the opportunity to provide their insight on project impacts.
- JICA’s Ethiopian agricultural development program involved community participation in the project from the earliest planning phases, with 100% of members reporting that they had “participated” or “actively participated” in the process. This resulted in feelings of greater personal ownership of the project, and heightened local understanding of their responsibilities.
- Evaluations that included direct community feedback in their analyses were by far the most helpful when trying to determine sustainability. For instance, in JICA’s Agricultural Development Project in the Kambia District of Sierra Leone there was no mention of local level involvement throughout any of the stages of project planning, implementation, or evaluation, which could have influenced why the project only “somewhat” achieved its objectives
- Projects that have flexible agendas, willing to change with the changing needs of the population during the planning/implementation phases, are viewed positively by the developing community and achieve more successful outcomes. This willingness to adapt was what characterized the project in GVC OLNUS Argentine Puna. Considering the true, up-to-date needs of the community allowed for greater local participation that enabled the strengthening of local autonomy (and thus, sustainability).
- None of the project evaluations provided a breakdown of how successful budget allocation was. The JICA projects included a breakdown of the overall budget into equipment and local costs, however despite some evaluations noting who provided certain funding, none mentioned if parts of the budget were inefficiently used. We believe it would be helpful to include not just how much money was invested in a project, but also how much of that budget either prompted direct growth or failed to produce an effective outcome.
Local community members are often referred to as ‘beneficiaries’ in the development process, yet they are the ones who governments, NGOs, and multilateral organizations are trying to empower through their various socioeconomic development missions. So, when we need to understand what worked with a project, and as importantly what didn’t work for a project, it is the voices of the community that need to be heard. A lot of great work is being done in international development, but it is clear that after her initial research that ex-post evaluations are essential to determining project sustainability and that projects that propose community-level development must also take the time to directly involve those community members in their own evaluation process. This feedback loop has the power to inform and influence future projects, while also creating the opportunity to actually listen to what participants (not beneficiaries) can sustain for themselves to achieve a better life.
Where have you found feedback loops that work? What excellent programming can you share?
Sources:
[1] Nishimaki, R., Kunihiro, H., & Tahashi, S. (2008, July 8). Evaluation Result Summary: The Project for Irrigation Farming Improvement. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/project/term/africa/c8h0vm000001rp75-att/ethiopia_2008_01.pdf
[2] Kumagai, M., Otsuka, M., & Sakagami, J. (2009, September 26). Evaluation Result Summary: The Agricultural Development Project in Kambia in the Republic of Sierra Leone. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/project/term/africa/c8h0vm000001rp75-att/ethiopia_2008_01.pdf
[3] The Improve Group. (2012, December). Post Project Evaluation of Mercy Corps’ MILK Program in Niger: Examining Contributions to Resilience. Retrieved from https://www.yumpu.com/en/document/read/35930718/niger-milk-post-project-evaluation-final-report-mercy-corps
[4] Proatec SRL. (2013, March). Ex Post Evaluation of Projects Managed by NGOs in Argentina. Retrieved from https://www.oecd.org/derec/italy/Evalutation-of-Projects-Managed-by-NGOs-in-Argentina.pdf
[5] International Fund for Agricultural Development (IFAD). (1997, August). Small-Scale Agricultural Development Project – Ex-post Evaluation. Retrieved from https://www.ifad.org/en/web/ioe/evaluation/asset/39828071
by Kelsey Lopez | Feb 6, 2014 | Evaluation, Local Participants, Participation, Results, Sustainability, Uncategorized
How do we define Sustainability?
Sustainability is a key success outcome for which a project is assessed, one of the five OECD DAC Criteria for Evaluating Development Assistance, which also includes relevance, effectiveness, efficiency, and impact. However, it is important to understand how this term, sustainability, is defined in a more specific way. Traditionally, the evaluation of project sustainability is linked primarily to the financial viability of the project into the future. Will there be an extension of donor funding to continue development activities or an operational budget for maintaining new technologies? Will there be sustainability of resources to pay for continued manpower and training programs? While the financial sustainability of a project is an important factor, Valuing Voices has a different approach to assessing sustainability that focuses primarily on the capacity of the community, rather than the donors, to achieving long-term program success. The Valuing Voices definition is:
Sustainability – assessed by looking at what communities can maintain themselves three, five, or ten years after a development program is completed. The focus is not on the financial sustainability of donor-funded activities, unless that is what communities request.
With this definition in mind, evaluations should be more heavily focused on attaining community feedback regarding what they believe were the aspects of the project that had the greatest significant outcomes. For a program to positively impact the community it was implemented in, it is crucial that the local community members themselves have the ability to sustain the activities that benefitted them the most, ideally with their heightened economic capacities. Implementing organizations and donor feedback is important as well, but without considering the opinions of the clients in developing societies, what do we really know? There are a few components to the evaluation process that are essential for the Valuing Voices model of sustainability to be accomplished:
1. Full Participation of community participants and local partners, at all stages of project design and implementation;
2. Planning for sustainability from the start, and ensuring that any knowledge resources generated by the project are designed to be accessible to communities;
3. Measure outcomes and impact sustainability of projects from two-five years after the end;
4. Build and fund local evaluation capacity;
5. Create feedback loops where learning is shared among and between participants, implementing partners, governments and other stakeholders at the local, national, regional, and global level;
6. Use technology to effectively capture data in a standardized format and facilitate feedback loops.
This is the central idea of the Valuing Voices mission, and now we’d like to hear feedback from YOU about what sustainability really means in the development context. Do you agree with this framework of analysis, or do you think there are other ways to define project sustainability? Does this definition changes in different contexts? Do you believe that the six components above capture the key elements for evaluating the sustainability of the project, or should other components be added? Let us know in the comments below so we can engage in an open discussion about sustainability – because everyone’s perspective (and voice) should be valued!
