Sustainability SPRINGing out all over the place… and Disrupting!

Sustainability SPRINGing out all over the place… and Disrupting

 

So what is sustainability? You may think it's the climate's long-term wellbeing and how to gauge changes to that.  You may think it's linked to sustainable development regarding consumption, trade, education and environment and how to assess it. You may think it's data-driven organizational success as Chelsea Clinton describes, or is it Michael Porter's business' view of Creating Shared Value on social and environmental concerns or is it about people, as hallowed University of Cambridge trains experts in its Institute for Sustainability Leadership (I revel that I was a Fellow there in the '90s). Finally, is it WCED’s lovely definition "Sustainable development is development that meets the needs of the present without compromising the ability of future generations to meet their own needs"? Yes, when applied to communities' abilities to self-sustainably and resiliently chart their own development! 

So how are we to get there? A Sustainable Brands Conference this year gets us there through being clear about their own consumption, and USAID is no different. USAID Forward is putting their money where their keyboards are (so to speak), toward more sustainable local delivery by directing a huge 30 percent of its funding to “local solutions” through procurement in coming years.  This framework is to “support the ‘new model of development’ that USAID Administrator Rajiv Shah has touted, which entails a shift away from hiring U.S.-based development contractors and NGOs to implement projects, and toward channeling money through host-country governments and local organizations to build their capacity to do the work themselves and sustain programs after funding dries up. I, and others celebrate the investments this will enable local firms to make in their own capacity, in leading development!

Of course all sorts of safeguards are needed, and ideally US firms would be providing capacity development, but shouldn’t we have been doing this all along, to move toward transferring ‘development’ to the countries themselves?

 GAO_Sustain_LocalSyst0514

Source: GAO report

Also vital to sustainable development is learning from what works and doing more of it. USAID is finally planning to incorporate more ex-post evaluations into its toolkit of evaluating sustainability!  Two weeks ago, PPL/LER shared their great new policy document- “Local systems: A framework for supporting sustained development on how they can better incorporate local systems thinking into policy as well as DIME (Design, Implementation, Monitoring and Evaluation).  Industry insider DevEx tells us "even though the agency plans to use ex-post evaluations to measure whether development projects are successful or not, these evaluations will not focus on “specific contractor performance” but instead consider the “types of approaches that contribute to more sustainable outcomes…to inform USAID’s country strategies and project design." While PVO implementing partners will not [yet?] be required to do ex-post evaluations as part of their projects, having this door cracked open is excitingly opening. Notably, it is a ‘back to the future’ moment, as 30 years ago USAID led the development world in post-project evaluations, yet in the last 24 years has done none (or at least not published any) except for the Food for Peace retrospective below, as I found in our Valuing Voices research of USAID's Development Experience Clearinghouse.

There is far more to watch. In our view, the whole development industry needs to grapple with the perceived barrier that funding ends with projects (note: a trust could be set up to document post-project impact 1, 3, 5 years later and results retained, much as 3ie does now for impact evaluations) and the view that one cannot discern attributable project impact with a time-lag of several years. Yet even the General Accounting Office is asking for longitudinal data; they reviewed USAID’s document and wants to see clear measures of success at Mission and HQ level by different indicators of local institutional sustainability and impact four years on.

Why should we care? As Chelsea Clinton of the Clinton Global Initiative puts it, "you can't measure everything, but you can measure almost everything through quantitative or qualitative means, so that we know what we're disproportionately good at. And, candidly, what we're not so good at, so we can stop doing that.

Yes! Development should be about doing more of what works, sustainably, and less of what doesn’t. USAID’s Local Systems Framework found the best could also be free, as in this one Food For Peace evaluation shows:

FFPBox8

Returning to Chelsea Clinton, I’ll conclude by stating something obvious. She "wants to see some evidence of why we're making decisions, as opposed to the anecdotes” which is what getting post-project evaluation data from our true clients, our participants, is all about. Clinton says this will transform CGI into a smart, accountable, and sustainable support system for philanthropic disrupters around the world. USAID is radical for me, today, with their Local Systems investments… my neighborhood disrupter.

 

Are you such a disrupter too? Who else is one whom we can celebrate together? 

Pineapple, Apple- what differentiates Impact from Sustainability Evaluation?

Pineapple, Apple- what differentiates Impact from Sustainability Evaluation?

There is great news.  Impact Evaluation is getting attention and being funded to do excellent research, such as by the International Initiative for Impact Evaluation (3ie), by donors such as the World Bank, USAID, UKAid, the Bill and Melinda Gates Foundation in countries around the world.  Better Evaluation tell us that “USAID, for example, uses the following definition: “Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other that the intervention that might account for the observed change.”

William Savedoff of CGD reports in Evaluation Gap reports that whole countries are setting up such evaluation institutes:  “Germany’s new independent evaluation institute for the country’s development policies, based in Bonn, is a year old.  DEval has a mandate that looks similar to Britain’s Independent Commission for Aid Impact (discussed in a previous newsletter ) because it will not only conduct its own evaluations but also help the Federal Parliament monitor the effectiveness of international assistance programs and policies. DEval’s 2013-2015 work program is ambitious and wide – ranging from specific studies of health programs in Rwanda to overviews of microfinance and studies regarding mitigation of climate change and aid for trade.” There is even a huge compendium of impact evaluation databases.

There is definitely a key place for impact evaluations in analyzing which activities are likely to have the most statistically significant (which means definitive change) impact. One such study in Papua New Guinea found SMS (mobile text) inclusion in teaching made a significant difference in student test scores compared to the non-participating ‘control group’ who did not get the SMS (texts).  Another study, the Tuungane I evaluation by a group of Columbia University scholars showed clearly that an International Rescue Committee program on community-level reconstruction did not change participant behaviors. The study was as well designed as an RCT can be, and its conclusions are very convincing.  But as the authors note, we don’t actually know why the intervention failed. To find that out, we need the kind of thick descriptive qualitative data that only a mixed methods study can provide.

Economist Kremer from Harvard says ““The vast majority of development projects are  not subject to any evaluation of this type, but I’d argue the number should at least be greater than it is now.” Impact evaluations use ‘randomized control trials’, comparing the group that got project assistance to a similar group that didn’t to gauge the change. A recent article that talks about treating poverty as a science experiment says “nongovernmental organizations and governments have been slow to adopt the idea of testing programs to help the poor in this way. But proponents of randomization—“randomistas,” as they’re sometimes called—argue that many programs meant to help the poor are being implemented without sufficient evidence that they’re helping, or even not hurting.”  However we get there, we want to know – the real (or at least likely)- impact of our programming, helping us focus funds wisely.

Data gleaned from impact evaluations is excellent information to have before design and during implementation.  While impact evaluations are a thorough addition to the evaluation field, experts recommend they be done from the beginning of implementation. While they ask “Are impacts likely to be sustainable?”, and “to what extent did the impacts match the needs of the intended beneficiaries?” and importantly “did participants/key informants believe the intervention had made a difference?” they focus only on possible sustainability, using indicators we expect to see at project end rather than tangible proof of sustainability of the activities and impacts that communities define themselves that we actually return to measure 2-10 years later.

PineappleApple

That is the role for something that has rarely been used in 30 years – for post-project (ex-post) evaluations looking at:

  1. The resilience of expected impacts of the project 2, 5, 10 years after close-out
  2. The communities’ and NGOs’ ability to sustain which activities themselves
  3. Positive and negative unintended impacts of the project, especially 2 years after, while still in clear living memory
  4. Kinds of activities the community and NGOs felt were successes which could not be maintained without further funding
  5. Lessons for other projects across projects on what was most resilient that communities valued enough to do themselves or NGOs valued enough to get other funding for, as well as what was not resilient.

Where is this systematically happening already? There are our catalysts ex-post evaluation organizations, drawing on communities’ wisdom. Here and there there are other glimpses of ValuingVoices, mainly to inform current programming, such as these two interesting approaches:

  • Vijayendra Rao describes how a social observatory approach to monitoring and evaluation in India’s self-help groups leads to “Learning by Doing”– drawing on material from the book Localizing Development: Does Participation Work? The examples show how groups are creating faster feedback loops with more useful information by incorporating approaches commonly used in impact evaluations. Rao writes: “The aim is to balance long-term learning with quick turnaround studies that can inform everyday decision-making.”
  • Ned Breslin, CEO of Water For People talks about “Rethinking Social Entrepreneurism: Moving from Bland Rhetoric to Impact (Assessment)”. His new water and sanitation program, Everyone Forever, does not focus on the inputs and outputs, including water provided or girls returning to school. Instead it centers instead on attaining the ideal vision of what a community would look like with improved water and sanitation, and working to achieve that goal. Instead of working on fundraising only, Breslin wants to redefine the meaning of success as a world in which everyone has access to clean water.

We need a combination. We need to know how good our programming is now through rigorous randomized control trials, and we need to ask communities and NGOs how sustainable the impacts are.  Remember,  99% of all development projects worth hundreds of millions of dollars a year are not currently evaluated for long-term sustainability by their ultimate consumers, the communities they were designed to help.  

We need an Institute of Sustainable Evaluation and a Ministry of Sustainable Development in every emerging nation, funded by donors who support national learning to shape international assistance. We need a sustainability global database, mandatory to be referred to in all future project planning. We need to care enough about the well-being of our true client to listen, learn and act.

Stepping up community self-sustainability, one [Ethiopian] step at a time

 

Stepping up community self-sustainability, one [Ethiopian] step at a time

 

Having just come back from evaluation and design fieldwork for an Ethiopian Red Cross (ERCS)/ Swedish Red Cross/ Federation of the Red Cross and Red Crescent project, the power of communities is still palpable in my mind. They know what great impact looks like. They know what activities they can best sustain themselves. It’s up to us to ask, listen and learn from them and support their own monitoring/ evaluating/ reporting. It’s up to us to share such learning with others and to act on it everywhere.

There are a myriad of possible sustainability indicators, and the outcome indicators below, suggested by 116 rural participants from Tigray, Ethiopia seem to fall into two categories of expected changes: Assets and Life Quality (Table 1). As the food security/ livelihood project extended credit for animal purchases, it is logical that tracking increased income, savings, assets, and home investments plus expenditures on food and electricity appeared.

We gleaned this from discussions with participants, asking them “what can we track together that would show that we had impact”? Our question led to a spirited discussion of not only what was traceable, but also what could be publicly posted and ‘ground-truthed’ by the community. Discussing indicators led to even deeper conversations about the causes of food insecurity which were illuminating to staff.  What was surprising, for instance, was the extent to which families saw changing seasonal child-field labor practices in favor of 100% child-school attendance as great indicators.  School attendance (or lack thereof) was dependent on families’ need for children’s seasonal labor in the fields. Community members said they knew who sent their children or not, which no only ‘cleaned’ the publicly posted data but triangulated implementer surveys and opened room for discussions of vulnerability.

 

self-sustain-indicators

 

 

Not only is this exciting for the project’s outcome tracking but even more importantly, our team proposed to create a community self-monitoring system, suggested in by Causemann/Gohl in an IIED PLA Notes article– “Tools for measuring change: self-assessment by communities” used in Africa and Asia. This learning, management and reporting process will fill a gaping need as current “monitoring systems serve only for donor accountability, but neither add value for poor people nor for the implementing NGOs because they do not improve effectiveness on the ground.” The authors found that not only “participatory data collection produces higher quality data in some fields than standard extractive methodologies [as] understanding the context leads to a higher accuracy of data and learning processes [which] increase the level of accountability… “ but also that such shared collaboration builds mutual learning and bridge-building.” While our community members may have offered to track this publicly to make this partner happy, men and women discussed this excitedly and embraced the idea of self-monitoring happily. ERCS will be discussing with communities to either track data monthly in notebooks or on a large chart hung in the woreda office for transparency.  Data (Chart 1) would include these asset and quality of life indicators as well as loan repayments (tracked vertically) while households (tracked horizontally) could see who was meeting the goal (checked boxes), not meeting it fully (dashed boxes) or not meeting it at all yet (blank boxes).  Community members corrected each other as they devised the indicators during our participatory research and this openness reassures us that the public monitoring will be quite transparent as well.

 

ParticipTracjking

[1]

Further, what was especially satisfying was getting feedback from across the three tibias (sub-regions) on what activities they felt they could sustain themselves irrespective of the project’s continuation. Table 2 shows us which activities communities felt were most self-sustainable by households; these could form the core of the follow on project. Sheet/goats, poultry and oxen for fattening were highly prioritized by both women and men, in addition to a few choosing improved dairy cows. The convergence of similar responses was gratifying and somewhat unexpected, as there were several other project activities.  The communities’ own priorities need to be seriously considered as currently they get only one loan per family and thus self-sustainable activities are key.

 

self-sustain-activities

 

There is more to incorporate in future project planning by NGOs like Ethiopia’s ERCS. The NGO-IDEAs concept mentioned above also includes involving project participants in setting goals and targets themselves, differentiating between who achieved them and why, and brainstorming who/what contributes to it and what they should do next. Peer groups, development agencies and any actors could collect and learn from the data. Imagine the empowerment were communities to design, monitor and evaluate and tell us as their audience!

And they must, according to ODI UK’s Watkins, who has a clear vision on how to achieve a global equity agenda for the post-2015 MDG goalsHe suggests converting the principle of ‘leave no one behind’ into measurable targets. He argues that, by introducing a series of ‘stepping stone’ benchmarks, the world can set ambitious goals on equity by 2030. He writes, wisely, that “narrowing these equity deficits is not just an ethical imperative but a condition for accelerated progress towards the ambitious 2030 targets. There are no policy blueprints. However, the toolkit for governments actively seeking to narrow disparities …has to include some key elements [such as] identifying who is being left behind and why is an obvious starting point. That’s why improvements to the quality of data available to policy-makers is an equity issue in its own right”. Valuing Voices believes who creates that data is an equally compelling equity issue.

 

So how will we reach these ambitious targets by 2030? By putting in stepping stone targets, returning project design functions to the ultimate clients – the communities themselves- and matching their wants with what we long to transfer to them. In this way we will be Valuing their Voices so much that they evaluate our projects jointly and we can respond. That’s how it should always have been.

What are your thoughts on this? We long to know.

 

 

Sources:

[1] Ashley, H., Kenton, N., & Milligan, A. (Eds.). (2013). Tools for supporting sustainable natural resource management and livelihoods. Participatory Learning and Action, (66). Retrieved from https://pubs.iied.org/14620IIED/

[2] Watkins, K. (2013, October 17). Leaving no-one behind: An equity agenda for the post- 2015 goals. Retrieved from https://www.odi.org/blogs/7924-leaving-no-one-behind-equity-agenda-post-2015-goals

 

What are the prospects for self-sustained impact at the Macro level? (Part 1)

What are the prospects for self-sustained impact at the Macro level? (Part 1)

After studying various impact evaluations, it became clear that different development projects also have different scales, with multilaterally-funded macro-level being focused on higher scale objectives from the municipal up to the national level while micro-level are geared towards improving socioeconomic aspects of communities. Our research revealed that there is a recurring error among projects that makes them ineffective: a lack of community involvement in all stages of the project, or more specifically, their lack of involvement in both the design and the evaluation process. This results in a lack of opinions from the local level regarding project self-sustainable goals, and prospects for ultimate success in self-sustainability. We have seen both macro projects, implemented by multilaterals, and micro projects, implemented by international non-profits, that fail to include local participation in their methodology, but one trend we have noticed is that projects on the macro level, have a tendency to plan, implement, and complete projects with little input from local participants or consideration for the possibility of self-sustainability impacts.

A particularly egregious example of non-self-sustainability at any level is the project for OECD-funded Improving the Performances and Management of Public Lighting in Ho Chi Minh City, which was a project with no ground level/community participation, but rather took place at the municipal level between French company Citelum (supplying technical assistance and equipment) and Sapulico (Saigon Public Lighting Company, Ho Chi Minh City's public lighting authority). From the evaluation summary, we saw the project was most concerned with propagating appreciation for French expertise in the area of public lighting at the national level rather than fostering national electrical self-sustainability. With organizational sustainability as a main objective, the project itself was not considered self-sustainable for a few key reasons:

  • “Sapulico's engineers acquired the necessary [skills required to operate the system], however, their current skill level is insufficient to handle present and future technical difficulties without the assistance of Citelum.” This means that the local company in Vietnam would continue to be reliant on foreign expertise in the likely event of future technical difficulties.
  • Also, jarringly, “there are also concerns even about the project's current viability as Sapulico's budget doesn't allow for the purchase of equipment from abroad, therefore the company will be unable to replace equipment that has reached the end of its lifecycle.”

ElectricalSystem

This highlights an obvious overlooking of future project sustainability due to the local budget conditions and insufficient training. Had the project been more concerned with guaranteeing even national self-sustainability rather than the financial sustainability of the foreign implementing organization, the project overall could have been more successful far into the future….

A more promising example of a macro level project aiming for more localized impact, yet still failing to consider self-sustainability, is the JICA Agricultural Development Project in Kambia District.  While we commend JICA as the most active multilateral to do ex-post evaluation, we again see the tendency of macro level projects to focus on systemic changes at the national or at least regional level which often self-sustainability buy-in, whereas micro level projects tend to focus on communities or at most sub-regions with the objective of fostering longer-term sustainability. This project, on the other hand, is a hybrid of macro- and micro- as it was to intervene at a district level and it did try to create agricultural extension buy-in. The key takeaways from JICAs post-project evaluation:

  • This project was concerned with improving agricultural productivity at select sub-district pilot sites, and then a further extension of the project throughout the entire Kambia District. The evaluation focuses heavily on the inability of the project to complete this objective of extending the project geographically because there was insufficient manpower, technical support, and funding to complete project dissemination, which suggests that sustained impact was not fully considered when planning implementation. It also suggests it did not expand because lessons seem not to have been learned from the implementation, and funding was not longer available.
  • The key point here is that the project did not secure adequate resources or funding for the true completion of its objective, which was to extend the successful outcomes that they observed at the pilot sites further across the region. This resulted in an evaluation analysis of sustainability that was only “fair”.
  • This project also fails to mention any local participation in the design, planning, implementation, or long-term sustainability phases, so despite seeing successful outcomes in increased crop production at the pilot locations, the evaluation was ultimately negative.
  • It is clear from the evaluation that the project was being executed from the side of the implementing agency rather than involving local participation beyond the extensionists, and no input was asked from community members about resulting agricultural self-sustainability.
  • The crux is that had the project been more focused on local self-sustainability beyond the regional extension system, its own organizational deficiencies (lack of manpower, technical support, and funding mentioned above) might not have hindered the ability to expand the project further.
     

Compare these findings to those we have analyzed in other blogs which illustrated the far more successful project and evaluation results that micro level non-profit development agencies like Mercy Corps, Plan International, and Partners for Democratic Change fostered when they made community participation the key to their development process. However, JICAs Kambia project and the Vietnamese Public Lighting program are not alone in their shortcomings, because the consistent lack of local community involvement in community-focused projects is a theme that we continue to see with macro level and micro level development schemes – a pattern that Valuing Voices is determined to change.

Overall, what we have discerned from this analysis is that projects conducted by macro level multilateral actors have a lower chance of ensuring self-sustainability, in part because they usually aren’t linked directly to the community in regional or national-scale projects. By contrast, micro level projects by international non-profits tend to have a higher chance of fostering greater self-sustainability because they are linked more closely to the end users of development, thereby benefiting both the organization itself as well as the local participants from more inclusionary practices.

We can clearly see that at least a few organizations, both at the macro and micro level, are asking the right questions by conducting these post-project evaluations. However, a systemic problem persists in that virtually no one is designing for self-sustainability. But without it, how successful are we really?