Prospects for long-term sustainability…or lack thereof at the Macro Level (Part 2)

Prospects for long-term sustainability… or lack thereof at the Macro level (Part 2)

 

During 2003-2004, the OECD conducted a very interesting Ex-Post Evaluation Sustainability Summary that synthesizes four separate Regional Rural Development (RRD) projects, which has illuminated sustainability of the impacts of German RRD projects over nearly a 30-year period.  The goal of this evaluation was to use the perspectives of local experts (evaluators and national consultants) to inform the Federal Ministry for Economic Cooperation and Development about the true impacts of four rural long-term projects.

 

The four projects evaluated were:

·      Indonesia – Area Development Project in West Pasaman (1980-1992); German contribution: €32 million; Target population: 200,000 urban and rural people

·      Sri Lanka – Regional Rural Development Project (RRDP) in Kandy (1987-2000); German contribution: €8.1 million; Target population: 200,000 rural small-scale producers

·      Tanzania – Tanga Integrated Rural Development Program (TIRDEP) (1972-1993); German contribution: €75 million; Target population: 700,000 rural small-scale producers

·      Zambia – Integrated Rural Development Program (IRDP) in Kabompo (1977-1993); German contribution: €30 million; Target population: 65,000 rural small-scale producers

 

While these programs “had been implemented in regions with very different underlying general conditions,” involving interventions of varying scales, with different socioeconomic conditions affecting the different countries and have effected a great number of changes in their respective countries, German’s RRD experts also drew similarly positive findings on outcomes and negative findings in terms of processes and the pitfalls of programming without focus on sustainability.

 

Here is the good news:
 

·      “Living conditions of the target groups have improved in all four project regions,” with specific sustainable project outcomes observed in the “health and education sector, food security, increase in income and employment and the ensuing rise in the standard of living.” The projects were able to create these changes by improving infrastructure, enhancing the private sector economy, and supporting innovations in agriculture.

·      Project planning was done largely on the German side, however it was determined that there was still a relatively high acceptance of the objectives by the participating stakeholders (locals) due to the establishment of joint objectives by locals and the implementing agencies.  

Yet, there were also some factors that jeopardized the sustainability of positive results, which included the (non)maintenance of the infrastructure as well as the intensification of economic activities that led to negative ecological impacts (such as acidification of the soil and overfishing).

looking backwards

Even more interestingly, the OECD gets kudos from ValuingVoices for analyzing where the projects failed to meet their outcomes and sharing what led to unsustainable outcomes. Here is the bad news:
 

·      There was low institutional sustainability at the level of state executing organizations for all four projects due to inadequate funds, inefficient organizational structures and a lack of coordination. Thus, viable exit and handover was limited. Due to this lack of institutional sustainability, it was concluded that, “the putting into place of new structures by means of development projects runs a very high risk of not being sustainable,” noting significant differences in local expectations of the partners and the concepts of the RRD. Hence imposing structures ex-post rather than designing with sustainability in mind doesn’t seem to work.

·      There is also an issue of changing standards over time. These projects were completed from the 1970s-1990s, but older development schemes are not considered as relevant today. “The former RRD project concept is no longer pursued” today as a result of changes, such as economic reforms and decentralization, which led to the adoption of new concepts. This kind of risk bedevils all ex-post evaluations, akin to asking perfectly good donkeys why they are not thoroughbreds.
 

Further, the processes this collection of ex-post studies illuminates are key:

 

·      “The expenses for the consultancy services of German experts were often considered as disproportionate compared with the hardware supplied. Here, partner expectations obviously were not consistent with those of the German side.” Potentially this is an incentive to use national evaluators. Yet the study concluded: “this serial evaluation has brought about a change in perspectives in part through the deployment of local experts. However, the qualification profile of locally available consultants varies considerable. The results of this serial evaluation do not suggest a general shift of evaluations to local organizations.” This study was 10 years ago so this has also begun to shift, even though pairing a ‘northern’ evaluator with ‘southern’ ones is more the norm.

 

·      Finally, “the termination of German support was often considered abrupt and incomprehensible. Phasing out was done according to German views and did not take sufficient account of the partners’ views and needs as seen by them…[which] provides grounds for a systematic conflict between the interests of the partners and those of the German side.”  This is key in reforming international development, from artificial timelines to those informed by adaptive learning, that takes into account and is even led by community and local NGO learning and readiness to take the helm themselves.

 

There is much to learn from OECD reviews such as this one about both the aims and processes for ex-post evaluation. What remains unclear in all ex-post evaluations Valuing Voices has found is what the organizations and donors have learned from them, and to what degree they have applied the lessons learned to the rest of their programming. 

What are your experiences?  Have lessons from evaluations been taken in? How? Why not?

 

Sustainable Development… are we done yet?

Sustainable Development… are we done yet?

 

 

 

 

 

The UN's Commission on Sustainable Development has just disbanded after 20 years. They have given their convening power and work to a High‐level Political Forum, saying "we must effectively use the new High‐Level Political Forum to ensure that sustainable development continues to be implemented and is integrated into the heart of the post‐2015 development agenda." 

Yes, while a few organizations have actually funded some post-project (ex-post) evaluations here and there, this idea that anyone in our industry could say "sustainable development continues to be implemented and integrated" strikes me as absurd. How many communities in Africa, Asia, Latin America and elsewhere feel that they are sustainably developed, that they can finally hand out gilded keys to their stunning cities and villages and never have to invite anyone in for help again?  Scant few, I'd wager (Botswana and Asian tigers notwithstanding). 

One of the fascinating findings of my preliminary research is that extremely few ex-post evaluations of sustained impact of projects are done at all, and the lions share is by some banks and multilateral organizations. While 30 years ago USAID used to fund such work, today it is UN agencies such as the OECD, the World Bank1 and World Bank2/ IMF, bilateral banks such as the European Investment Bank, IADB, a German Bank, and a few bilateral development organizations such as Japan's JICA.  Kudos to them for doing it at all, anytime. 

Why do some do it? An Australian government publication notes that:
* "Good ex post evaluation is a potentially powerful tool that can attest to the value of a reform and provide lessons for future reforms. 
* Evaluations can improve the understanding of the impacts of regulation on risk — 
both the probability of an adverse event and the impact if it arises."

This publication also states that "to be more effective, ex post evaluations need to: be resourced in advance; embed data collection where it is not otherwise available; share the findings; and include governance arrangements which support transparent process to test the findings and to encourage their use."  The bit about use is especially compelling, especially if such results are shared with all stakeholders, especially national ones- from communities involved, up to policy decision-makers. Better yet,  start discussions about sustainability with local communities to begin with, where rubber hits the road.

Unfortunately, reading some of these reports is a bit surreal; it shows how far from that end-user perspective we currently are.  While evaluators ask questions about long-term impact, questions of sustainability often remain 'within the box' of expected outcomes. Reports I have read use an external evaluator who alone assesses OECD's measures of: program relevance, effectiveness, efficiency, development impact and sustainability of the project as it was planned.  So far nothing has appeared that implies that participants themselves were asked about sustained impact, rarely are traces of anything 'outside the box' of what was planned (such as the appearance of unintended impacts), or that anything could have happened outside of the project objectives in the elapsed years post-project. This is in spite of our understanding that our world is complex and our lives and livelihoods adaptively evolve all the time in response to external economic, climatic, social opportunities and shocks.  "Sustainability" results seem to be reported as if time stopped after the end of project (often many years after the project was planned, the results framework crafted), and little evolved in these countries to affect project sustainability.

Finally, the rose-colored glasses aspect includes (the possible fiction) that those involved in the project shared the Goals and Objectives aspirations held by the donors and implementers. In my fieldwork experience, communities sometimes cannot tell one project from another unless one helps them differentiate specific resources such as wells, seeds, literacy training transferred. Nor do participants necessarily share expectations of outcomes such as they would be more drought-resilient or food-secure in the future.  In research I hope to do, I will look for how many shared expectations of impact there are, how communities evaluate long-term impact of the tangible and intangible resources that were transferred and what their perceptions of the real value of projects is, long after implementers and donors have left.

 

What do you think? Have you seen cases where we are doing Sustainable Development well?