Learning from a river of ex-post project evaluations and tools… Thanks USAID!
Dear ex-post aficionados. It’s raining ex-post project evaluations. Here’s hoping learning from such evaluations in water/ sanitation, maternal/child health and even capacity building/ peacekeeping, and their number increases!
1. WATER/ SANITATION & HYGIENE:
USAID has a series of six ex-post evaluations of the water/ sanitation and hygiene sectors since 2017! What is exciting is that they are also looking to the future. These evaluations will “provide insight into what happens after an activity ends, and how to mitigate challenges in future programming, potentially. The series will inform USAID’s WASH activity design and implementation and contribute to a larger sector discussion on achieving sustainability.”
The E3 water division (Water CKM ) took sustainability on as their strategy and have made great strides these last two years. They have done five ex-post project evaluations, cited below, and MSI has completed one more wat/san/ hygiene ex-post evaluations, specifically:
Madagascar Rural Access to New Opportunities for Health and Prosperity (RANO-HP) – Published June 2017
The first evaluation in the series explores the sustainability of the sanitation and hygiene components of the RANO-HP activity, implemented in 26 communes from 2009–2013.
Indonesia Environmental Services Program (ESP) – Published August 2017
The second evaluation in the series examines the sustainability of water utility capacity building, microcredit, and financial outcomes associated with the ESP activity, which was implemented from 2004–2010.
Ethiopia Millennium Water Alliance (MWA-EP) – Published May 2018
The third evaluation in the series examines the long-term sustainability of outcomes related to rural water point construction, rehabilitation, and management, as well as participatory sanitation and hygiene education and construction related to the MWA-EP activity, implemented in 24 rural districts between 2004–2009.
Financial Institutions Reform and Expansion–Debt and Infrastructure (FIRE-D) – Published September 2018
This evaluation is the fourth in the series. It examines how urban water and sanitation services in India have changed since FIRE-D closed and to what extent policies, practices, and financing mechanisms introduced through FIRE-D have been sustained.
Millennium Water and Sanitation Program in Senegal (PEPAM/USAID) – Published July 2019
The fifth ex-post evaluation in the series looks at the PEPAM project (Programme d’Eau Potable et d’Assainissement du Millénaire au Sénégal), implemented from 2009–2014 to improve sustainable access to WASH in four regions of Senegal.
USAID-funded by MSI: USAID/Ghana’s Water Access, Sanitation, and Hygiene for Urban Poor (WASH-UP)– published Nov 2018
Also USAID and Rotary International developed a WASH Sustainability Index Tool, “to assess a WASH activity’s likelihood to be sustainable according to the following factors: availability of finance for sanitation; local capacity for construction and maintenance of latrines; the influence of social norms; and governance.” This is similar to what we learned from USAID/ FFP/ Tufts/ FHI360 12 ex-posts that resources, capacities, motivation and linkages (aka partnerships, including governance) are vital to sustaining outcomes and impacts.
It will be interesting to see whether they examine the other ex-posts for excellent lessons, as they have the Senegalese evaluation:
- “Whether or not to subsidize sanitation access …Based on this evaluation’s findings and exploration of the literature, subsidies can help improve the quality of household latrines, but increasing use of those latrines remains a challenge.
- In contrast, CLTS (a nonsubsidized approach) is often credited with increasing use of unimproved latrines, but serious questions linger about quality and long-term sustainability of the latrines built after CLTS triggering, particularly as it relates to moving up the sanitation ladder. This evaluation… provides the opportunity to examine the potential value of a hybrid approach….
- The handwashing results suggest that low-cost, low-quality handwashing stations such as tippy taps do not lead to sustained behavior change. It may be worth considering hygiene investments that reduce the behavior change burden on targeted beneficiaries.
2. MATERNAL/ CHILD HEALTH & NUTRITION:
“Sustainability of a Community-Based CHOICE Program to Improve the Health and Nutrition Status of Mothers and Infants in Indonesia,” The report focused on whether the USAID-funded CHOICE program had left sustainable impacts: improving the health and nutrition status of children under the age of five, as well as the health status of pregnant and lactating women and mothers or caretakers of young children in the Pandeglang District of Indonesia. “After examining the data collected from the PSS, the researchers found that there were significant improvements in many indicators—such as births attended by skilled personnel, the treatment of diarrhea, and the nutritional quality of food fed to infants—in the six years after the CHOICE program ended. However, despite these improvements, the researchers found no significant statistical differences between villages that received the CHOICE program interventions and comparison villages, which did not. This speaks to using such a comparison methodology to focus on actual contribution and rule out the “rising tide lifts all boats” phenomenon.
3. CAPACITY DEVELOPMENT & PEACEBUILDING:
USAID’s Regional Office in Thailand evaluated its capacity building and peacebuilding program 1.5 years ex-post. While civil society was strengthened and there were inroads made on peacebuilding, “many interventions initiated during Sapan did not continue post-Sapan, although some did remain. For example, “stakeholders cite evidence of continuing to use some governance tools in local governance related to service delivery [although] because of limited financial resources after Sapan ended, they had to change some of their interventions and reduce the range of people they could include. There are lessons for whose capacities are built, two-way feedback loops with local partners, using local organizations such as universities to sustain training, planning sufficient time for partners to internalize training lessons, etc.
4. USAID FUNDED GUIDANCE:
‘Impact Evaluations’ have a new focus on long-term impact, rather than effectiveness during implementation (which was at least the original intent of impact evaluation in the 1980s)! In September 2018, USAID and Notre Dame issued a Guide for Planning Long-term Impact Evaluations as part of the Utilizing the Expertise of the ERIE Program Consortium. The guide covers the difference between traditional impact evaluation designs and data collection methods and how to apply them to long-term impact evaluations (LTIE). It also shares examples across a range of sectors, including later evaluating past impact evaluations, which ended before final evaluation.
Finally, in new 2018 USAID guidance, ex-post evaluation is clarified as the source of the sustainability of services and benefits. USAID clarifies that “questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation, a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be, and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project’s services and benefits were sustained.”
Such richness that we can learn from. Keep the momentum going on the 99% of all global projects yet unevaluated ex-post, and change how we fund, design, implement, monitor and evaluate global development projects!
The Name of the Game is “Sustainability” but Does the Last Player Count?
by John Lowrie, Reblog from https://www.scribd.com/document/11621086/The-Name-of-the-Game-is-Sustainability-but-Does-the-Last-Player-Count-by-John-Lowrie
Today, it is obligatory to answer the question “how will your proposed activities be sustainable after the project?” Most of us dutifully play the game and repeat various sentences that describe the measures that “should” bring sustainability about. Then it is usually left at that. We all move on to the next project and donor. Seldom does anyone look back over the passing of many years to check to see if promise has turned in to reality. At best there may be an end-project evaluation which will say that planned outcomes are achieved and “likely” to be sustained. It would be a very brave evaluator to be more committed beyond that.
My question is why do we play this game? (I should add: apart from the obvious answer we need the donors’ money!) Sustainability is much more than a ruse in a game. It goes to the very essence of what kind of organisation we belong to; plus all the others with us whether staff, supporters or beneficiaries; how we originated and what are our real long- term plans?
Many NGOs are artificially created groupings of people, unlike the first ones last century that emerged as people on a mission to address specific issues that they felt strongly about. Latterly, NGOs are now lumped in to “Civil Society” which is another recent fashionable label in development jargon. This is despite many officials and even NGO workers in Cambodia having no idea that it is supposed to encompass more than NGOs, to embrace other groups such as the press, trade unions, etc.
Many NGOs have actually been started by or for charismatic individuals, tapping in to somebody else’s cause or source of funding, rather than part of a collective if not mass movement in its own right towards a common end. Sometimes, international departing NGOs promote them, as part of “exit-strategies” to demonstrate that they leave something behind to show for their good work and all the money spent. They may have even included a plan for it in their sustainability proposal write-up. In Cambodia the “gravy train” that accompanied
UNTAC, the massive international effort to bring lasting peace and democracy in the early 90s, spawned many. Some of these NGOs have been good and stood the test of time. Many have fallen by the wayside. Others have been co-opted by political interests. Few have proper accountable self-governance structures.
◄“It’s the money that matters”, but will these NGO beneficiaries, the final players, get what they need to be sustainable?
So how can they be really sustainable? What is driving them – apart that is from the obvious – money? It is true that many do good work. They often do work that local authorities should be doing. But, I ask, towards what end? For example “poverty alleviation” is not an end in itself; for it to be sustained it needs much more than the usual 1-3 year time-frames that donors favour for their projects. Yet the usual pattern is –identify your target groups, go there, pass on whatever to them, then move on (to the next ones). There are exceptions. Lutheran World Federation (LWF) expect to work for 10 or more years in their target communities before they “graduate” and begin a systematic staged withdrawal. Mostly, however, the projects end on time or soon after, the files are closed, and that is that, at least until next time.
Meanwhile NGOs try to source repeat funding or new funding, and the game rules dictate where and how they proceed. It is only by luck rather than design that there will be a true match between what they want to do and what the donor is willing to give the money for. Usually there is accommodation, most likely on the part of the NGO; indeed some big donors explicitly rule out comments, queries, and changes to their guidelines which become grails of holiness. The most desperate NGOs have to re-invent themselves; depart from their original mission, suddenly acquire new skills in often far removed fields to stay in the game. I have seen an election-monitoring organisation become an agency to consult displaced families affected by a road-widening. I know of one local rural development NGO in one province become a pro-citizen governance campaigner in another. Adaptability and learning to diversify are good qualities, but when they cause such radical changes within an NGO, it cannot be re-assuring for sustainability.
The nature of what an NGO does and its underlying philosophy is therefore key to sustainability. Those NGOs created to work in the immediate post-conflict or disaster emergency relief periods are prone to short-term visions, and sometimes they leave legacies which handicap development such as “dependency” and even “easy-come-easy- go” attitudes when foreign money seems plentiful. Those NGOs born in the next phase, i.e., after emergency relief, the start of infrastructure reconstruction and restoring public services and the economy, can also be equally short-term in their vision. In fact they have an inherent flaw. If they succeed, they do themselves out of a job and few want that as it would mean no future jobs and income! Even the Cambodian Government, despite an addiction to foreign aid, now maintains that some have overstayed their welcome.
These NGOs tend to be “welfare” or “service-provision” providers. They are confident in their abilities “we know best” but are they committed to passing their best skills and knowledge on and to the right people to take progress forward without them?
Cambodia is not alone in that politics plays a big part. The ruling party now has a 73% majority in the National Assembly, holds 98% of the 1,629 Commune Council Chiefdoms and 70% of Councillors. As the national and commune members form the constituency for the Senate, Provincial, and District Authorities, the party is guaranteed monopoly control. The situation is not helped by the absence of a neutral civil service or public service – in fact it is mystifying that UNTAC and every international donor has not tried to cultivate one. So we are left with what we have; we are where we are. It means that sustainability will only follow if we accept that reality. We have to engage constructively with the powers-that-be; we have to find the best people we can to work with, and through them, reach out to others. If not, eventual opposition or just plan lack of good will and support will affect the final outcome. Authorities always outlast NGOs, at least in Cambodia they do!
Cambodia did not have a good start when NGOs first came on the scene. Many of the first NGOs were human rights activists needed at that time and to a certain extent now, to expose calamitous treatment meted out to victims. Unfortunately this profile, still high in perceptions, has made many in power believe that this is only or mainly what NGOs do and stand for1. The legalistic approach to human rights where abuses are reported, perpetrators identified, and “name, blame shame” attached, cannot be a development tool in the sustainability tool-kit. It must be separated out and equally important alternative human rights approaches such as “rights-based development” which arouse less hostility co-exist with similar enthusiasm and means. NGOs and civil society will never have sustainable activities while their undoubted overall positive contribution gets little or no recognition by the people that count most when it comes to change.
NGOs are agents of change, which if mishandled does lead to suspicion, so to achieve change, NGOs must be clear in their message to persuade all (or a majority) that their change is worthwhile. If a long-term, wide cross-section commitment towards that change is not implanted, it is not sustainable. Too many development initiatives in Cambodia have resulted in temporary change. The “status-quo” reverts soon afterwards. In some cases the change is only tacitly accepted “take their money”, “go along with them” but “once they have gone, we go back to how we were, OK?” This is the opposite of sustainability.
There have to be contextually appropriate solutions, which can only be country-by- country, culture-by-culture. We should not have to operate on the basis of big international donors with their “one-size-fits-all” development policies and calls for proposals that allow just one format (theirs) that automatically favours the big international NGOs. They have their professional fund-raisers, who can knock out proposals that score high marks in the assessment/evaluation boxes, without many of the authors and assessors concerned ever going near the intended target beneficiaries! Yet be under no doubt, the words they pen and peruse answer beautifully on how such beneficiaries were involved, but who ever checks?
I may not be a good team-player in the present development game, so what am I suggesting as an alternative? First of all I would like to see fewer NGOs, ones which are smaller, more self-contained, and manageable to operate country-by country, sector-by- sector, region-by-region. They need to have good links to others elsewhere for best practice to be shared, but their core mission needs to be focussed to bring about certain defined changes; with the right people and the resources they need, and in the fullness of time. Therefore 5 years is an absolute minimum and asking for “core costs” to be provided should not be regarded as a mortal sin, as it tends to be with most donors. It has to be allowed, to be seen as value for money, and though indirect, still an essential element in bringing about that change. If not, how can sustainability be served?
Ironically, it is only with core support or independent means, that NGOs can play the game, so again favouring the big players. How else does an NGO cover its costs in preparing bids to donors? For example after UK-DfiD released their worldwide Global Transparency Fund in 2008, 450 organisations applied, and just 38 succeeded. Even a recent in-country release by UNDP for environmental projects attracted no less than 67 applications, of which just 13 won out. This means that for 412 in the DfiD case and 54 in the UNDP case who had devoted considerable effort, on top of their normal work, the process ultimately proved to be a waste of time; to be disappointing and especially for local NGOs to be discouraging.
I have two (NGO) organisations in mind while writing this article. Neither conforms to the pattern I criticize. They are different and I am not the only one to think that they are better. Both have poor disabled people as their target beneficiaries. One is in sports. The other is in poverty alleviation/new livelihoods or careers and self-advocacy. Both are now purely local NGOs, with none of the trappings or spending power of the big international players that dominate the disability sector. In fact neither has anything like a long-term future, because of funding gaps, and so a lot of their work, despite excellent results so far, cannot be said to be sustainable. If they go out of business too soon, how will their beneficiaries stay involved, if sustainability is to be realised eventually?
Local NGO “Cambodia National Volleyball League Disabled2” is the sports one. Most interestingly, it is taking a route party by choice, partly by necessity, towards the private sector or “corporate social responsibility” funds in its hope to realise sustainability. CNVLD has earned a worldwide reputation for transforming the self-esteem of disabled athletes. They actually enjoy high standing on the world stage unlike their “able-bodied” compatriots. They play volley-ball, pursue wheel-chair racing and some athletes may well qualify for the 2012 London Paralympics, even if there may be no money to support them to go and compete there.
◄Disabled Sports Athletes of NGO CNVLD playing the game at its best – but do they have a future?
Now CNVLD’s quest is to raise such money and cover their modest (compared to INGO) costs. Yet this laudable aim is treated with derision by some in the sector. Pursuing private funding is viewed as an anathema to many who see it as contradictory to the “not- for-profit” concept. Does this make sense? Does such obstruction make for sustainability? Is it a sin not to want to depend solely on the usual institutional sources of funding? One critic is interesting. It has in its mission that all its services to beneficiaries must be provided free. No charges whatsoever must be levied even for those who can afford to pay! In Cambodia, everybody pays, even when services are supposed to be free3. Even if a direct charge is not levied, recipients expect and are expected to show their gratitude4, and do so especially the poorest who see it as an inescapable obligation. Surely sustainability means (a) people who can afford to pay should do so, and (b) NGOs who can attract funding and not depend on taxpayers’ money should be welcomed5?
The second organisation, New Horizons Society6, is not as lucky as CNVLD, as they cannot go down the route of sponsorship which is an accepted feature of sport. This is an NGO that did localise from an INGO but on their own terms. They voted against joining another national body created by and favoured by the big international disability NGOs. Over 200 of their Focus Persons (Group Leaders) voted in a secret ballot to form their own NGO in order to stay true to their close-knit grassroots upwards origins and growth. Now they have 3,175 members in 135 self-help groups federated up to provincial level and going on to the national stage. They have remarkable accomplishments in creating new livelihoods for their [once] ultra-poor members and have accumulated over $130,000 in revolving funds. Individual lives have been transformed. One boy has gone from beggar in the market to national singing celebrity. One young man went from lonely at home; never seen a computer; no English, to being one of today’s high-flying geeks. His class-mate, from a similar start, went on to become the Publicity Officer for CNLVD and to lead her own troupe of dancers in the NHS Child Advocacy Group performing at international conferences.
◄NGO Child Advocates demonstrate “We can do” but will donors let them?
The 135 groups went from fear of talking to officials to successful advocacy. They started with the right to education and health-care for disabled youngsters, and once they were confident went on to persuade ministers to take action to stop their meagre pension rights being denied to them. These people chose not to ask for pity, or welfare, or for service provisions to be given to them. Instead they ask simply for the chance to show that “they can do” and that they can be self-sufficient when given the opportunities and means.
Their problem is that they cannot do this for all members yet, let alone go on to include others in the same fate as they once were. Yet despite their accomplishments, right now they can only win project activity funds. Donors refuse to pay more than 20% for running costs. Some specify as little as 5%. That does not even cover the running costs of their multi-purpose centre where meetings, training, sports, dancing, computer classes etc., go on. It may stretch to pay modest salaries to 2 or 3 staff, but they have to depend on consultant/advisors like me to help them voluntarily. Their entire organisation is radically different from the familiar set-up to be seen in international disability, donor and development organisations. There is not an air-conditioner, land-cruiser, or voucher paying university fees of expatriate’s children, etc., in sight. Yet despite its low cost and high yield, it is not yet sustainable within the present rules of most donors. If it can get over its current shortfalls and continue to build the revolving funds in to a sizeable trust or investment fund, with many members whose incomes have grown sufficiently able to subscribe fees to run their organisation, then they may have an independent viable future. But is there a donor who can adopt such a long-term vision; who will give enough to meet the real needs they identify, and who will then stay the course even when inevitable setbacks happen on the way?
So finally sustainability should be expressed in one simple notion and by the last player. It should be a measure of the change made in individual lives and over life-times of beneficiaries and their families. They are the last players in the game – who else but them can make that assessment? The present game means that it is the other players that have most say. They arrive on the scene much earlier; play in their own compact time-frames, and to their own rules. Then as external entrants, they depart the scene as soon as they can. Sustainability?
1 The perception is not helped by the limitations of the Khmer language. When the word “advocacy” was first introduced “tasumateh” was used, literally “to struggle for” associated with confrontation, not partnership, and so it was viewed by many as an alien Western concept.
2 email@example.com or www.standupcambodia.net
3 Understanding pro-poor political change:the policy process Cambodia by Caroline Hughes and Tim Conway, Overseas Development Institute, 2003 – DfiD Publication.
4 It is this “tradition” that is at the heart of a current dispute within the UN-backed Khmer Rouge
Trials, where Cambodian staff are alleged to have paid a proportion of their salary to officials with a role in their appointment.
5 Subject of course to disclosure and transparency.
6 firstname.lastname@example.org or www.newhorizonsunlimited.org