The EU and Sustainability
At the Prague European Summit I was invited to last week, sustainability was on the program three times. The most relevant session was Towards a More Sustainable and Prosperous European and Global Economy through Trade and Investments, pictured below.
I asked Panelists like Sabine Weyand, Director General for Trade at the European Commission, to define Sustainability. She defined it as actions the EU is taking, including mutually reinforcing Twin Transitions (Green and Digital) to foster a carbon-neutral EU by 2050 and the three pillars of sustainability, namely:
- 1st pillar is ‘climate and environment,
- 2nd, a zero carbon economy (including biodiversity), and
- 3rd, social sustainability (equity by member states and internationally).
These were followed by several panelists mentioning the issue of member countries needing to support and fund these. One would think that funding would be gratefully accepted and reciprocated given the astonishing €338bil the EU already spent across Europe, eight times the size of the Marshall Plan.
The panelists of these sessions sketched out ambitious plans. I researched this oft-promised “EU Green Deal” to understand its scale, if the sustainability of results (which I work on ex-post), the sustainability of our global ecology, and/or sustainability in terms of business functions resides, if at all. The results of this cursory research did not make these clear. But ambitions are massive.
EU Green Deal
The European Green Deal began when the EU Parliament adopted the EU Climate Law in 2021, which makes “legally binding a target of reducing emissions 55% by 2030 and climate neutrality by 2050. This moves the EU closer to its post-2050 objective of negative emissions and confirms its leadership in the global fight against climate change.”
SIDENOTE1: Given that our lawsuit against the Czech Republic government for not meeting its Paris Agreement commitments just failed and the Ministry of the Environment successfully claimed that reducing emissions by only 26% was enough (rather than the EU’s target of 55%), I wonder how much of these ‘legally binding’ targets will actually be met…
Turning to what I could find of the 1st and 2nd pillars (as the 3rd was missing from Green Deal online documents which I found other than mentions of a ‘Just Transition’):
- AID FUNDING (SDGs)
The EU as a leading global partner for the funding the SDGs & Paris Agreement
The EU seems to be shifting from the SDGs which appear to be bilaterally funded, although ‘collectively’ claimed. “The EU and its Member States are the leading donors of official development assistance (ODA) globally. In 2022, they collectively provided €92.8 billion (based on preliminary OECD figures), which accounts for 43% of global assistance.” So what progress is that generating? Not much:
SIDENOTE2: And since we’re talking about climate, can the EU or other donors aid-fund functioning weather stations? They have 5% of the number the US and EU do which massively limits mitigation: https://africanarguments.org/2023/11/without-warning-africa-lack-of-weather-stations-is-costing-lives/
- SUSTAINABLE FINANCE FOR CLIMATE, ENVIRONMENT, AND ECONOMY (the 1st 2 pillars of the mentioned Sustainability)
So, for the EU, where I now live most of the time, sustainable finance plans seem to be off to a good start. Nearly 2/3 of the promised €300 billion has been secured in only two years. Notably, this is 1/3 of the EU’s planned €1 Trillion, including private investments. Two German policy researchers, Findeisen and Mack, question their achievability. “Europe needs to spend an additional €350 billion on climate action every year until the end of this decade…to reach its 55% greenhouse gas reduction target by 2030.” Furthermore, “shortcomings of [Investment Plan] InvestEU in combatting climate change can be addressed and why it is no substitute for fresh public spending at EU level.” (See the excellent brief including concerns about the Sustainable Europe Investment Plan, outlined below. This overview of the EU’s Investment plan has broadly planned outputs (e.g., ambitions, financing, research, mobilizing), outcomes (e.g., energy generated, building new energy sources, restoring ecosystems, and building food systems and mobility), and impacts (e.g., a sustainable future):
Much of the plan is, understandably, European-focused, yet we all know that the Global South is where the deepest needs and opportunities lie. Drilling down further to see investments, activities, and even results on the ground was both interesting and harder:
“The inaugural milestone of the Global Gateway was the Africa-Europe Investment Package with approximately €150 billion of investment dedicated to bolstering cooperation with African partners. We have also started implementing Global Gateway in Asia and the Pacific and in Latin America and the Caribbean, where President von der Leyen announced a global investment by the EU and its Member States of over €45 billion. In 2023, ninety key projects were launched worldwide across the digital, energy, and transport sectors through Global Gateway to strengthen health, education, and research systems globally.”
So, as an Africanist and evaluative researcher, I looked into the EU’s reported figures (and results). Under Global Gateway, the EU has funded 33 projects in Sub-Saharan Africa, 11 of which are climate-related and another 7 that are transport-related. There are only short descriptions of projects without substantive partners, activities, or data, such as a €1billion “Climate Change Adaptation and Resilience in Africa“. Details would be instructive, to say the least. Research on African Voices regarding their projects unearthed one that should already be of concern: Namibia’s $10 billion Green Hydrogen project, which is to help 2.5 million people decarbonize. However, the tendering is not transparent, locals are in the dark, there are biodiversity concerns, and initial funding is missing. Much more transparent monitoring & evaluation data is needed to know it is on track.
I also consulted African Voices regarding COP28 that are relevant to the EU’s Africa investments.
- Funds are needed. Lorraine Chiponda. a Coordinator of the Africa Movement Building Space notes that “Africa receives a meager 3% of the total global climate finance and African countries still play a marginal role in the global finance system, making it a mammoth task to obtain funds for renewable energy investment.”
- Africa has much to offer. Joseph Nganga is Interim Managing Director at the Global Energy Alliance for People and Planet (GEAPP) reminds us that “Africa, a region with a developing economy endowed with abundant natural resources, stands at a crossroads where strategic financial investments can steer its trajectory towards a prosperous climate-resilient future.”
- Moreover, financing needs are clearly expressed, which the EU, for one, should listen to. “This year at the African Climate Summit, countries were united in their call for a series of finance reforms that would have a meaningful impact on their fiscal and policy space to address climate change. These include a reduction in borrowing costs and risk premiums; debt management, restructuring and relief; scaling concessional climate finance from multilateral developments banks (MDBs); and reforms to the global tax regime.” Olivia Rumble is a climate change legal and policy expert and a director of Climate Legal
Much more is needed for equitable partnerships to address our increasingly desperate climate needs. While I was at the conference, the Copernicus Climate Change Service issued a warning that “For the first time ever, the planet globally exceeded a key warming threshold on Friday for the first time since at least the beginning of instrument records, new data shows. Simply put, a 2-degree rise in global temperatures was considered as a target for the end of the century and is considered a critical threshold above which dangerous and cascading effects will occur.” COP28 has a lot to accomplish.
Helpful? Let me know in the comments…
Evaluating global aid projects ex-post closure: Evaluability, and how-to via Sustainability (and Resilience) Evaluation training materials:
How do we evaluate projects ex-post, and are all projects evaluable after donors leave? How do we learn from project documents to ascertain likely markers of sustainability (hint: see materials for Theory of Sustainability and Sustainability Ratings) that we can verify? How do we design projects to make sustainability more likely (hint: implement pre-exit the drivers in the second image, below)? How to consider evaluating resilience to climate change (hint: evaluate resilience characteristics)?
In 2017, Valuing Voices created an evaluability checklist for Michael Scriven’s foundation, as many projects are not implemented long enough, or too long ago, or have such weak monitoring and evaluation (M&E) data as to be very difficult to evaluate: https://valuingvoices.com/wp-content/uploads/2017/08/Valuing-Voices-Checklists.pdf. We have used it in multiple evaluations since. Now in 2023, we are sharing work we’ve done with the Adaptation Fund, where we have applied that evaluability process via a vetting evaluability process that eliminated over 90% of all projects done in their early years, which alone is sobering but typical in our field, and is leading to changes in how they monitor, retain information and learn. See page 24 – officially pg 15) of this Phase 1 report: https://www.adaptation-fund.org/wp-content/uploads/2021/10/2021-09-12-AF-TERG_Ex-Post-Phase-1-Final-Report_Final.pd
Ex-post evaluability at the Adaptation Fund
The Adaptation Fund has also committed to doing two ex-post sustainability and resilience evaluations per year. and learn from remote ex-posts as well. The first two ex-post evaluations are found here: Samoa/ UNDP on resilient infrastructure (https://www.adaptation-fund.org/wp-content/uploads/2023/01/Ex-Post-Evaluation-Samoa-final-ed-2.pdf) and Ecuador/ WFP on food security (https://www.adaptation-fund.org/document/ex-post-evaluation-summary-ecuador/). So many lessons from assumptions made to future design…
Not only are we evaluating projects ex-post, but we are also creating processes for others to follow, including this new Sustainability Framework. The left three columns project the likelihood of sustainability, which the right three columns verify. It includes the Valuing Voices ’emerging outcomes’ from local efforts to sustain results after donors leave. SO exciting:
Ex-post Sustainability Framework
Also, just today, we published our training materials which we are using in ex-posts 3 and 4 in Argentina (Ministry of the Environment and the World Bank); https://www.adaptation-fund.org/document/training-material-for-ex-post-pilots/. They include videos for those who like to listen and watch us, as well as Powerpoint slides/ PDFs and Excels of our training materials (including suggested methods) for those who like to read…
Take a look, use, and tell us what you think and what you are learning! Thanks, Jindra Cekan (Sustainability) and Meg Spearman (Resilience) and the Adaptation Fund team!
Inter-American Development Bank (IDB) – where have your ex-post evaluations, and learning from them, gone?
A Linkedin colleague, Gillian Marcelle, Ph.D. recently asked me about ex-posts by the Inter-American Development Bank (IDB) as more Caribbean accelerators/incubators were planned without learning from previous identical tech investments. Here is what I found, and if anyone knows more, please contact me, as it is not reassuring. Also, some were internal ‘self-evaluations’, some were desk reviews, and only a few involved going to the field to ask aid recipients about what lasted, which is typical for multilaterals (ADB and the IBRD do the same). Given Valuing Voices’ focus on participant’s voices in results, there was an attempt to focus on those, but this report did not make it clear which were which so highlights are presented below.
In 2003 IDB created an ex-post policy, “Ex Post Policy (EPP) in October 2003, which mandated two new tasks to OVE: the review and validation of Project Completion Reports and the implementation of ex post project evaluations.”. These were under the Board’s request for “a commitment to a ‘managing for results’ business model.” 2004’s ex-posts were seen as “the first year of the implementation of the EPP, all 16 evaluations can be considered part of the pilot and the findings presented in this report refer to the entire set of ex post evaluations.” Further, “the general evaluative questions proposed by EPP are first “… the extent to which the development objectives of IDB-financed projects have been attained.” and second “… the efficiency with which those objectives have been attained.”
They spent over $300,000 unsuccessfully evaluating six of the projects. In part this was due to data quality. “six had an evaluation strategy identified in the approval stage, most had abandoned the strategy during execution prior to project closure and, with one exception which produced data that could be used to calculate a treatment effect, none had produced quality evaluative information…. No [Project Completion Report] PCR provided adequate information regarding the evolution of development outcomes expected from the project or an update with respect to the evaluation identified at the time of approval. For the other six, they found that the expected results did not match what would be the sustained results. Some were better than expected while more were worse. “A critical finding across all projects is the lack of correspondence between the reflexive estimates and the treatment effect estimates. In practically all cases, the estimates were different.” How sustainably “Improved” are “Lives” as IDB’s logo touts?
In 2004 IADB chose 16 projects to evaluate and dropped four for a variety of reasons. The remaining dozen projects ex-post evaluated were on land development, improving neighborhoods, and cash programming. There were data quality and comparability issues from the onset. In the land [tenure] ‘regularization’: “Six of the projects mention ex post evaluations in loan proposals, but none have been completed to date. OVE was successful in retrofitting a subset of outcomes expected for three projects: an attrition rate of 50%.” The neighborhood improvements had positive and negative results, with ‘retrofitting’ being needed regarding data. For the four evaluated, the overall conclusion was mixed. Both, that the projcts led to “greater coverage of certain public services.” and for two cases, “this impact was more pronounced for the poorest segments included in the treated population.” Nonetheless, much more was unachieved. “Beyond this, very little else can be said. The impact on the objectives related to human capital formation and income were not demonstrated. In the case of health interventions, perhaps the intervention type most directly linked to sanitation services; there has been no demonstrated link between the interventions and outcomes, even for the poorest segments of the beneficiary population. There was also no consistent evidence showing an increase in variables related to housing values.”
Regarding cash programming, there were individual evaluations that showed promise but only after statistical analysis of a control group, something which is sorely lacking in most foreign aid evaluations. An IDB project in Panama “shows that in some cases the reflexive evaluation, in fact, understated the true program treatment effect. The development outcome of this project was the reduction in poverty. A reflexive evaluation (the gross effects) of the incidence of poverty suggested that not only was the project unsuccessful but that it actually contributed to worsening poverty; the opposite of its intent. However, a treatment effect evaluation (the net effect) that compared “similar municipalities” shows that municipalities benefiting from FIS funds had a significant decline in poverty relative to comparable municipalities that did not receive FIS financing; the project had clear positive development outcomes”.
The IDB staff consulted in 2005 about the results “questioned whether the analysis of closed projects that were not required to include the necessary outcomes and data at the time of approval was a cost-effective use of Bank resources” which may be a reason why the Bank decided against doing more, in spite of many ex-post findings contradicting expected results. Astonishingly, since then, only one summary of a Jordanian ex-post in 2007 was found, but it is questionable that it is an actual ex-post closure. At a minimum, one would expect that the Bank would ensure data quality improved, and planned strategies would actually be done.
Finally, presumably a bank cares about Return on Investment. As a former investment banker, I would be concerned about the lack of learning, given the low cost of such learning versus the discrepancies found between expected and actual sustainability. Specifically, the extremely low cost of the six evaluations that were done ($113K each, more precisely a cost of .001-.21% of the program value), which is a pittance compared to the millions in loan values. Given that more were not done – or at least publically shared- in the 18 years since, sustainability-aware donors, beware.
Unlike this multilateral, I’ve been busy with two ex-post evaluations which I hope to share in the coming months… Let me know your thoughts!
Hard-wiring and Soft-wiring in Sustainability via health program examples: by Laurence Desvignes and Jindra Cekan/ova
We all want things to last. Most of us joined the ‘sustainable development’ industry hoping our foreign aid projects not only do good while we are there but long afterward. Following on last month’s blog on better learning about project design, implementation, and M&E, here are some things to do better.
Long-term sustainability rests on four pillars: the first rests on how the project is designed and implemented before exit and the second is to what degree conditions are needed for the continuation of results the project generated are put into place. While the first one embeds sustainability into its very results, the second invests in processes to foster the continuation of results. The other two of the four pillars, returning to see what lasts by evaluating the sustainability of results two or more years later and bringing those lessons back to funding, design, implementation, and building in shock-resilience, e.g., such as to climate change, are in other Valuing Voices blogs.
We focus on 1 and 2 in this blog, and use an analogy of hard-wiring and soft-wiring sustainability into the fabric of the project:
- Hardwiring, ‘baking-in’ sustainability involves the design/ implementation which predisposes results lasting. This includes investing in Maternal Child Health and Nutrition’s first 1000 days from conception to age two that are vital for child development. The baby’s physical development and nutrition are so important as is maternal well-being. Investing in these early days leads to better health and nutrition throughout their lives. So too are buying local. Too often our projects rely on imported technology and inputs that are hard to replace if broken. A project on hand pumps used by UNICEF suggested local purchase of those “designed to optimize the chances of obtaining good quality hand pumps and an assured provision of spare parts” involves both the hardware of the pump and also the “capacity building plan and a communication strategy.” Also using local capacity/specialists when available vs external consultants can also be key to building the sustainability of a project.
Another example of baking-in sustainability is using participatory approaches to ensure that those implementing- such as the communities/ local authorities. In this example, it’s grassroots where participants are heard during design in terms of their priorities and how the project should be implemented. This includes targeting discussions and monitoring and evaluation being done with and by communities. The seminal research of 6,000 interviews with aid recipients, Time to Listen, found that they want to participate and when they do, things are more likely to be sustained, rather than being passive recipients…. there is ex-post proof such programming is more ‘owned’ and more sustained.
Conducting in-depth needs assessments at design is usually the way to collect information about what is needed and how projects should be implemented to last. Unfortunately, very often, the time is very limited for the proposal development and (I)NGOs are under pressure of short deadlines to submit the proposal, and needs assessments are either done quickly, collecting very basic information or not done at all. Yet time spent valuing the voices of participants can bring great richness. In 2022, the UN’s FAO did a monitoring and evaluation study in Malawi validating indicators for poverty by asking communities how they identify it from the start. “Researchers were impressed at how accurately the people they interviewed were able to gauge the relative wealth of their neighbors.” We were not surprised as the locals often know best. Another example with Mines Advisory Group in Cambodia, we developed a community-based participatory approach for design whereby project staff would work with the mine-affected communities to draw local maps of their villages, highlighting the location of the dangerous places and the key areas/places used by the communities. Staff and communities discussed the constraints, risks, needs, etc. to make their community safer, which the project would follow up with risk education, clearance, victim assistance, and/or alternative economic /development solutions to make the community safer. Other mine action agencies, e.g. Danish Refugee Council (Danish Demining Group) are also now using safer community approaches, involving local residents to decide on how to make their village safer depending on the community priorities.
Hardwiring in participatory feedback-loop learning from locals during implementation is also key. Implementation of a community feedback strategy once the programme is running is also essential. The community feedback mechanism (CFM) is a formal system established to enable affected populations to communicate information on their views, concerns, and experiences of a humanitarian agency or of the wider humanitarian system. It systematically captures, records, tracks, and follows up on the feedback it receives to improve elements of a response. CFM is key to ensuring that people affected by crisis have access to avenues to hold humanitarian actors to account; to offer affected people a formalized structure for raising concerns if they feel their needs are not being met, or if the assistance provided is having any unintended and harmful consequences; to understand and solicit information on their experience of a humanitarian agency or response; as part of a broader commitment to quality and accountability that enables organizations to recognize and respond to any failures in response; to promote the voices and influence of people affected so their perspectives, rights, and priorities remain at the forefront of humanitarian/development work.
Promoting and implementing community engagement, such as a community feedback strategy, provides a basis for dialogue with people affected on what is needed and on how what is needed might best be provided, especially as needs change during implementation. This will help identify priority needs and is a means to gauge beneficiaries’ understanding of activities being carried out, to assist in the identification of local partners and establishment and follow-up of partnerships, and in the organizational development and capacity building of local institutions and authorities. It can strengthen the quality of assistance by facilitating dialogue and meaningful exchange between aid agencies and affected people at all stages of humanitarian response and result in the empowerment of those involved. Targeted people are viewed as social actors that can play an active role in decisions affecting their lives.
OXFAM’s project in Haiti starting in 2012 came as a result of a cholera epidemic that began in 2010 (“Preventing the Cholera Epidemic by Improving WASH Services and Promoting Hygiene in the North and Northeast”), whose goal was to contribute to the cholera elimination, experimented with the community feedback strategy as a means of gauging the recipients’ understanding of the activities carried out and of further strengthening the links between OXFAM and the communities during implementation. The initial process of community feedback was intended to both receive recommendations from project participants for better management of the action and also to better understand the strengths and weaknesses of Oxfam interventions. Based on the information and recommendations applied, OXFAM served as a bridge between the community and the actors involved (e.g. private firm contracted to carry out some health centers/ water systems renovation work or other) in the implementation of the project. This is also part of Oxfam’s logic of placing more emphasis on the issue of accountability and community engagement.
The feedback-loop benefits of such a community process are manyfold, especially on Protection, human rights, risk management, and further below, adapting Implementation, M&E, and fostering organizational learning:
- CFMs assist in promoting the well-being, rights, and protection of people by offering them a platform to have a voice and be heard
- it fosters participation, transparency, and trust
- It uses Do No Harm and conflict-sensitive programming
- It helps identify staff misconduct
- It functions as a risk management and early warning system
Adapting Implementation and Improving M&E:
- This process makes it possible to adapt to the priorities of the beneficiaries, to better meet their needs hence ensuring the agency’s accountability to the affected population
- It facilitates and guarantees a better quality of the project.
- It represents a means of monitoring our approaches and our achievements.
- It makes it possible to construct a common vision shared between the various actors and the project participants/targeted communities.
- Ensuring the programme quality and accountability through the establishment of an appropriate accountability strategy (including Transparency, Feedback, Participation, Monitoring, and Effectiveness) and relevant methodologies and tools (since the planning stage of the project) is a key exercise which allow to think and plan for the sustainability of the programme at an early stage.
- It allows us to gauge the strengths and weaknesses of the interventions while offering us the opportunity to learn from our experiences, hence allowing for programmatic learning and adaptative programming.
- It conveys the impact of the project and the change brought about in the lives of the beneficiaries.
- It is part of the logic of capitalizing on experiences to improve the quality of future projects.
2. Soft-wiring is creating conditions to make sustainability more likely for local communities and partners by thinking about how to replace what has been brought by the projects’ donors and implementers. This involves an analysis as well as actions that put conditions for sustainability into place before and during the time that foreign aid projects close. Valuing Voices’ checklists for exiting sustainably involves local ownership, sufficient capacities, and resources, viable partnerships, how well risks such as climate and economic shocks were identified and managed, and benchmarking for success 1-2 years before closure. Later it is important to return to check findings at ex-post project, comparing completion results to what was sustained 2-30 years later.
There are four categories of sustainability-fostering actions to do pre-exit which were identified by Rogers and Coates of Tufts for USAID for sustained exit:
Several blogs on Valuing Voices deal with resources, including assumptions donors make. Donor resource investments cannot be assumed to be sustained. The checklists outline a wide array of questions to ask during design and latest a year pre-exit, including what assumptions do aid projects make? USAID water/ sanitation/ hygiene investments have mostly not been sustained, due to a combination of lack of resources to maintain them and low ownership of the resources invested. Some key questions are:
- Did the project consider how those taking over the project would get sufficient resources, e.g., grant funding or other income generation available or renting out their facility or infrastructure that they own or shift some of their activities to for-profit production, sold to cover part of project costs?
- Does the project or partner have a facility or infrastructure that they own and is rentable to increase resources outside donor funding or can the project shift to for-profit, including institutional and individual in-kind products or technical knowledge skills that can be sold to cover part of project costs?
- What new equipment is needed, e.g. computers, vehicles, technical (e.g. weighing scales) for activities to continue, and which stakeholder will retain them?
- Or even no resources are needed because some project activities will scale down, move elsewhere, focus on a smaller number of activities that are locally sustainable, or the whole project will naturally phase-out)
The objective of that Oxfam project was to reduce the risks of communities placed in a situation of acute vulnerability to the cholera epidemic in two departments in Haiti (where about 1.5 million inhabitants reside). It focused on sustainability by effectively supporting and accompanying governmental WASH and health structures in the rapid response to alerts and outbreaks recorded in the targeted communities. How? Through awareness-raising activities among the populations concerned, by strengthening the epidemiological surveillance system and coordination between concerned stakeholders. The project also aimed to improve drinking water structures such as drinking water distribution points, drinking water networks or systems, catchments, and boreholes. As part of this intervention, Oxfam worked in close collaboration and in support of the Departmental Directorates of Health (DH), DINEPA (government services responsible for water and sanitation), and local authorities at the level of cities, towns and neighborhoods, and community structures including civil protection teams. Oxfam and DINEPA staff intervened through mixed mobile response teams that included technical and managerial staff from the health department to whom Oxfam provided ongoing technical support in terms of WASH analysis and actions, WASH training, finance training, and monitoring, as well as logistical support for the deployment of teams in the field (provision of vehicles and drivers). Oxfam was therefore working to ensure that cholera surveillance and mitigation actions were led by state and community actors, and by supporting state structures to build their capacities and allow ownership of the various aspects of the fight against cholera. Concretely, this was done as follows:
- Preliminary meetings and discussions were held with concerned governmental authorities to agree on a plan of action based on needs, implementation means, priorities, and budget for the health and wash governmental services/teams to be able to function. This was followed by the signature of an MoU between Oxfam and the Departmental Directorates of Health (DH).
- An action plan was set up with the DH and DINEPA (governmental water and sanitation agency) at the very beginning of the project.
- Outbreak response teams were managed directly by the DH and the staff was recruited, managed, and paid by the DH. The DH and DINEPA implemented the activities, managed the staff of the mobile teams, and provide technical monitoring in coordination with Oxfam.
- The epidemiological monitoring activities carried out by the DH were also monitored by the Oxfam epidemiologist who, in close coordination with the DH, built the capacities of epidemiologists and staff at the departmental level and at the level of the treatment centers to ensure adequate monitoring and communication.
- An Oxfam social engineering officer worked with DINEPA to ensure that the various water committees at the sources/infrastructure rehabilitated by Oxfam were functional. Sources/infrastructures were rehabilitated in concert with DINEPA to ensure the proper ownership.
- Oxfam provided funding, and technical supervision and wrote and submitted the final report to the donor. Based on DH’s regular reports on activities which were then consolidated by Oxfam for the donor.
- Teams were paid directly by the DH from funds received by Oxfam, based on the budget agreed by both Oxfam and the DH, and were based on government salary scales.
- The Oxfam WASH team, which systematically accompanies case investigations in the field, further encouraged the participation of DINEPA and its community technicians, through regular meetings with the DINEPA departmental directors.
- Overall, Oxfam ensured to provide support and capacity building of the DH, DINEPA, and community actors involved in the fight against cholera, to ensure proper ownership and to avoid substitution of the health/wash authorities.
The type of peer-partnering at design and during implementation described above is vital for ownership and sustainability. Unless we consider people’s ownership of the project and capacities to sustain results, they won’t be sustained. See Cekan’s exiting for sustainability checklists on phasing over before phasing out and exit, strengthening ownership, which brings us full circle to the participatory hard-wiring described above in Haiti.
We have to strengthen capacities at the most sustainable level. Taking an example from IRC’s Sierra Leone Gender-based Violence project involves looking at what happens when capacities training done for local participants and partners to take over are not done right. In this case, there were two-year consultancies to the Ministry (MSWGCA) on strategic planning and gender training, but “it is not clear if this type of support has had a sustainable impact. The institutional memory often disappears with the departure of the consultant, leaving behind sophisticated and extensive plans and strategies that there is simply no capacity to implement.” The report found that community-based initiatives that are the “primary sources of support for GBV victims living in rural areas in a more innovative and sustainable way that promotes local ownership. They also may yield more results,” most donor agencies find it hard to partner with community-based organizations so they recommended a focus on training and capacity-building of mainstream health workers to respond to GBV and aim for the government will assume control of service provision in approximately five years. The excellent manual by Sarriot et al on Sustainability Planning, “Taking the Long View: A Practical Guide to Sustainability Planning and Measurement in Community-Oriented Health Programming puts local capacity strengthening at the core. We have to consult and collaborate throughout and create an ‘enabling environment’ so that the activities and results are theirs.
Source: Sarriot et al 2008
Obviously, we should check on the sustainability we hope for. As ITAD/CRS note, we should do and learn from more ex-post evaluations which is much of what Valuing Voices advocates for.
Recommendations for fostering sustainability:
Few donors require information on how hard-wired or how soft-wired programming pre-exit is at closure which would make sustainability likely. Even fewer demand actual post-closure sustainability data to confirm assumptions at exit, sadly we believe most of our foreign aid has had limited sustained impacts. But this can change. Donors need to be educated that the “localization” agenda is the new trend (just as gender, resilience, and climate change have been at one point). It is beyond the “nationalization” of staff members (e.g. replacing expatriates with national staff), which is only one of the elements relating to locallization. True localization is to promote the local leadership of communities in their own ‘sustainable development’. While this is easier to say than to do, sustainability depends on it. We foster it through the hard-wiring and soft-wiring we discussed above and more steps, below. Here are specific steps from Laurence’s and Jindra’s experiences with the Global South:
- Funds & additional time for local partnership and ownership need to be embedded in the design and planned for, which requires a different approach on which the donors also need to be sensitized/ educated/ advocated to;
- In-depth needs assessment must be carried out just before or when an NGO sets up an operation – it usually takes time and should be integrated into any operation. Advocating this approach to donors is key so that it can be included in the budget or the NGO needs to find its own funds to do so) and the NGO country and sector strategy can then be updated yearly to embed such activities into the (I)NGO DNA;
- Conduct a capacity strengthening assessment of the local authorities or partners with whom we are going to conduct the project. This can take between 3 to 6 months, depending on the number and type of actors involved but this is an essential element to build self-sustaining local capacities and ensure that comprehensive capacity building is going to take place. This transparent step is also an essential step to ensure ownership by national/governmental stakeholders;
- It is vital to allow time to plan for an exit strategy at an early stage, even as early as design. This requires time and needs to be included in the budget for the implementation of the plan at least one year before the end, for phasing over to local implementing partners to take over before the donors/ Global North implementers exit, and for possibly strengthening capacities or extending programming to deliver on their timeline rather than ours before exiting out. More on this from CRS’ Participation by All ex-post and of course the oft-cited “Stopping As Success: Locally-led Transitions in Development” by Peace Direct, Search for Common Ground, and CDA. Also do not forget shared leadership noted by UK’s INTRAC’s “Investing in Exit”;
- Finally, don’t forget about evaluating ex-posts and embedding those lessons into future design/ implementation/ monitoring and evaluation.
Investing in sustainability by hard-wiring or soft-wiring works! Let us know what you do…
THE CONTENTIOUS POWER OF EVALUATIONS
or why sustainable results are so hard to come by….
A while ago, I reacted to a discussion among development aid/cooperation evaluators about why there are so few NGO evaluations available. It transpired that many people do not even know what they often look like, which is why I wrote a kind of Common Denominator Report: only about small evaluations, and self-explanatory in the sense that one understands why they are rarely published. The version below is slightly different from the original.
Most important elements of a standard evaluation report for NGOs and their donors; about twenty days of work; about 20,000 Euros budget (TVA included).
In reality, the work takes at least twice as much time as calculated and will still be incomplete/ quick, and dirty because it cannot decently be done within the proposed framework of conditions and answering all 87 questions or so that normally figure in the ToR.
The main issues in the project/ programme, the main findings, the main conclusions, and the main recommendations, presented in a positive and stimulating way (the standard request from the Comms and Fundraising departments) and pointing the way to the sunny uplands. This summary is written after a management response to the draft report has been ‘shared with you’. The management response normally says:
- this is too superficial (even if you explain that it could not be done better, given the constraints);
- this is incomplete (even if you didn’t receive the information you needed);
- this is not what we asked (even if you had an agreement about the deliverables)
- you have not understood us (even if your informants do not agree among themselves and contradict each other)
- you have not used the right documents (even if this is what they gave you)
- you have got the numbers wrong; the situation has changed in the meantime (even if they were in your docs);
- your reasoning is wrong (meaning we don’t like it);
- the respondents to the survey(s)/ the interviews were the wrong ones (even if the evaluand suggested them);
- we have already detected these issues ourselves, so there is no need to put them in the report (meaning don’t be so negative).
Who the commissioning organisation is, what they do, who the evaluand is, what the main questions for the evaluators were, who got selected to do this work, and how they understood the questions and the work in general.
In the Terms of Reference for the evaluation, many commissioners already state how they want an evaluation done. This list is almost invariably forced on the evaluators, thereby reducing them from having independent status to being the ‘hired help’ from a Temp Agency:
- briefings by Director and SMT [Senior Management Team] members for scoping and better understanding;
- desk research leading to notes about facts/ salient issues/ questions for clarification;
- survey(s) among a wider stakeholder population;
- 20-40 interviews with internal/ external stakeholders;
- analysis of data/ information;
- processing feedback on the draft report.
In the Terms of Reference, many commissioners already state which deliverable they want and in what form:
- round table/ discussion of findings and conclusions;
- draft report;
- final report;
- presentation to/ discussion with selected stakeholders.
PROJECT/ PROGRAMME OVERVIEW
Many commissioners send evaluators enormous folders with countless documents, often amounting to over 3000 pages of uncurated text with often unclear status (re authors, purpose, date, audience) and more or less touching upon the facts the evaluators are on a mission to find. This happens even when the evaluators give them a short list with the most relevant docs (such as grant proposal/ project plan with budget, time and staff calculations, work plans, intermediate reports, intermediate assessments, and contact lists). Processing them leads to the following result:
According to one/ some of the many documents that were provided:
- the organisation’s vision is that everybody should have everything freely and without effort;
- the organisation’s mission is to work towards having part of everything to not everybody, in selected areas;
- the project’s/ programme’s ToC indicates that if wishes were horses, poor men would ride;
- the project’s/ programme’s duration was four/ five years;
- the project’s/ programme’s goal/ aim/ objective was to provide selected parts of not everything to selected parts of not everybody, to make sure the competent authorities would support the cause and enshrine the provisions in law, The beneficiaries would enjoy the intended benefits, understand how to maintain them and teach others to get, enjoy and amplify them, that the media would report favourably on the efforts, in all countries/ regions/ cities/ villages concerned and that the project/ programme would be able to sustain itself and have a long afterlife;
- the project’s/ programme’s instruments were fundraising and/ or service provision and/ or advocacy;
- the project/ programme had some kind of work/ implementation plan.
This is where practice meets theory. It normally ends up in the report like this:
Due to a variety of causes:
- unexpectedly slow administrative procedures;
- funds being late in arriving;
- bigger than expected pushback and/ or less cooperation than hoped for from authorities- competitors- other NGOs- local stakeholders;
- sudden changes in project/ programme governance and/ or management;
- incomplete and/ or incoherent project/ programme design;
- incomplete planning of project/ programme activities;
- social unrest and/ or armed conflicts;
The project/ programme had a late/ slow/ rocky start. Furthermore, the project/ programme was hampered by:
- partial implementation because of a misunderstanding of the Theory of Change which few employees know about/ have seen/ understand, design and/ or planning flaws and/ or financing flaws and/ or moved goalposts and/ or mission drift and/ or personal preferences and/ or opportunism;
- a limited mandate and insufficient authority for the project’s/ programme’s management;
- high attrition among and/ or unavailability of key staff;
- a lack of complementary advocacy and lobbying work;
- patchy financial reporting and/ or divergent formats for reporting to different donors taking time and concentration away;
- absent/ insufficient monitoring and documenting of progress;
- little or no adjusting because of absent or ignored monitoring results/ rigid donor requirements;
- limited possibilities of stakeholder engagement with birds/ rivers/ forests/ children/ rape survivors/ people in occupied territories/ murdered people/ people dependent on NGO jobs & cash etc;
- internal tensions and conflicting interests;
- neglected internal/ external communications;
- un/ pleasant working culture/ lack of trust/ intimidation/ coercion/ culture of being nice and uncritical/ favouritism;
- the inaccessibility of conflict areas;
Although these issues had already been flagged up in:
- the evaluation of the project’s/ programme’s first phase;
- the midterm review;
- the project’s/ programme’s Steering Committee meetings;
- the project’s/ programme’s Advisory Board meetings;
- the project’s/ programme’s Management Team meetings;
Very little change seems to have been introduced by the project managers/ has been detected by the evaluators.
In terms of the OECD/ DAC criteria, the evaluators have found the following:
- relevance – the idea is nice, but does it cut the mustard?/ others do this too/ better;
- coherence – so so, see above;
- efficiency – so so, see above;
- effectiveness – so so, see above;
- impact – we see a bit here and there, sometimes unexpected positive/ negative results too, but will the positives last? It is too soon to tell, but see above;
- sustainability – unclear/ limited/ no plans so far.
If an organisation is (almost) the only one in its field, or if the cause is still a worthy cause, as evaluators you don’t want the painful parts of your assessments to reach adversaries. This also explains the vague language in many reports and why overall conclusions are often phrased as:
However, the obstacles mentioned above were cleverly navigated by the knowledgeable and committed project/ programme staff in such a way that in the end, the project/ programme can be said to have achieved its goal/ aim/ objective to a considerable extent.
Galileo: “Eppur si muove” = “And yet it moves”
Most NGO commissioners make drawing up a list of recommendations compulsory. Although there is a discussion within the evaluation community about evaluators’ competence to do precisely that, many issues found in this type of evaluation have organisational; not content; origins. The corresponding recommendations are rarely rocket science and could be formulated by most people with basic organisational insights or a bit of public service or governance experience. Where content is concerned, many evaluators are selected because of their thematic experience and expertise, so it is not necessarily wrong to make suggestions.
They often look like this:
Project/ programme governance
- limit the number of different bodies and make remit/ decision making power explicit;
- have real progress reports;
- have real meetings with a real agenda, real documents, real minutes, real decisions, and real follow-up;
Project/ programme management
- review and streamline/ rationalise structure to reflect strategy and work programme;
- give project/ programme leaders real decision making and budgetary authority;
- have real progress meetings with a real agenda, real minutes, real decisions, and real follow-up;
- implement what you decide, but monitor if it makes sense;
- consult staff on recommendations/ have learning sessions;
- draft implementation plan for recommendations;
- carry them out;
Processes and Procedures
- get staff agreement on them;
- commit them to paper;
- stick to them – but not rigidly;
Obviously, if we don’t get organisational structure and functioning, programme or project design, implementation, monitoring, evaluation, and learning right, there is scant hope for the longer-term sustainability of the results that we should all be aiming for.