Reblog: ITAD/CRS “Lessons from an ex-post evaluation – and why we should do more of them”

Reblog: ITAD/CRS “Lessons from an ex-post evaluation – and why we should do more of them”

Reposted from: https://www.itad.com/article/lessons-from-an-ex-post-evaluation-and-why-we-should-do-more-of-them/

Even as evaluation specialists, rarely do we get the chance to carry out ex-post evaluations. We recently carried out an ex-post evaluation of Catholic Relief Services’ (CRS) Expanding Financial Inclusion (EFI) programme and believe we’ve found some key lessons that make the case for more ex-post evaluations.

We’ll be sharing learning from the evaluation alongside CRS colleagues at the Savings Led Working Group session on Members Day of the SEEP Annual Conference – so pop by if you would like to learn more.

What is an ex-post evaluation?

Ex-post evaluations are (by definition) done after the project has closed. There is no hard and fast rule on exactly when an ex-post evaluation should be done but as the aim of an ex-post evaluations is to assess the sustainability of results and impacts, usually some time will need to have passed to make this assessment.

A little bit about EFI

EFI was a Mastercard Foundation-funded program in Burkina Faso, Senegal, Zambia and Uganda whose core goal was to ensure that vulnerable households experienced greater financial inclusion. Within EFI, Private Service Providers (PSPs) formed and facilitated savings groups using CRS’ Savings and Internal Lending Communities (SILC) methodology, with the SILC groups responsible for paying the PSP a small fee for the services that they provide.

This payment is intended to improve sustainability by incentivising the groups’ facilitators to form and train new groups, as well as providing continued support to existing groups, beyond the end of the project.

A little bit about the evaluation

So, if the aim of the PSP model is sustainability, you need an evaluation that can test this! Evaluation at the end of project implementation can assess indications of results that might be sustained into the future. However, if you wait until some time has passed after activities have ended, then there is much clearer evidence on which activities and results are ongoing – and how likely these are to continue. Uganda was also a great test case for the evaluation because CRS hadn’t provided any follow-on support.

Our evaluation set out to assess the extent to which the EFI-trained PSPs and their SILC groups were still functioning 19 months after the programme ended and the extent to which the PSP model had contributed to the sustainability of activities and results.

What the ex-post evaluation found

We found a handful of findings that were only possible because it was an ex-post evaluation:

  • There were 56% more reported groups among the sampled PSPs at the time of data collection than there were at the end of the project.
  • Half of the PSP networks established within the sample are still functioning (to some extent).
  • PSPs continued to receive remuneration for the work that they did, 19 months after project closure. However, there were inconsistencies in frequency and scale of remuneration, as well as variation in strategies to sensitize communities on the need to pay.

This only covers a fraction of the findings but we were able to conclude that the PSP model appeared to be highly sustainable. The evaluation also found that there were challenges to sustainability which could be addressed in future delivery of the PSP model. Significantly, the PSP model was designed with sustainability in mind – and this evaluation provides good evidence that PSPs were still operating 19 months after the end of the project.

What made the evaluation possible

We get it. It isn’t always easy to do ex-post evaluations. Evaluations are usually included in donor-implementer contracts, which end shortly after the project ends, leaving implementers without the resources to go back and evaluate 18 months later. This often results in a lack of funding and an absence of project staff. This is also combined with new projects starting up, obscuring opportunities for project-specific findings and learning as it’s not possible to attribute results to a specific project.

In many ways, we were lucky. Itad implements the Mastercard Foundations Savings Learning Lab, a six-year initiative that supports learning among the Foundation’s savings sector portfolio programmes – including EFI. EFI closed in the Learning Lab’s second year and with support from the Foundation and enthusiasm from CRS, we set aside some resource to continue this learning post-project. So, we had funding!

We also worked with incredibly motivated ex-EFI, CRS staff who made time to actively engage in the evaluation process and facilitate links to the PSP network, PSPs and SILC group members. So, we had the people!

And, no-one had implemented a similar PSP model in supported districts of Uganda since the end of EFI. So, we were also able to attribute!

Why we should strive to do more ex-post evaluations

Despite these challenges, and recognising it isn’t always easy, doesn’t mean it is not possible. And with projects like EFI where sustainability was central to its model, we would say it’s essential to assess whether the programme worked and how the model can be improved.

Unfortunately, practitioners and evaluators can shout all we like but the onus is on funders. We need funders to carve out dedicated resource for ex-post evaluations. This is even more important for programmes that have the development of replicable and sustainable models at their core. For some projects, this can be anticipated – and planned for – at project design stage. Other projects may show promise for learning on sustainability, unexpectedly, during implementation. Dedicated funding pots or call-down contracts for ex-post evaluations are just a couple of ways donors might be able to resource ex-post evaluations when there is a clear need for additional learning on the sustainability project results.

This learning should lead to better decision making, more effective use of donor funds and ultimately, more sustainable outcomes for beneficiaries.”

Other Findings:

Some of the other findings of this report on Financial Inclusion are:

RESOURCES: “Finding 1.iii. PSPs continue to receive remuneration for the work that they do; however, there are inconsistencies in frequency and scale of remuneration, as well as variety in strategies to sensitize communities on the need to pay.”

CAPACITIES: “Finding 3.ii. All networks included a core function of “collaboration, information-sharing and problemsolving”; however, networks were not sufficiently supported or incentivized to fulfill complex functions, such as PSP quality assurance or consumer protection, and their coverage area and late implementation limited the continued functioning of networks.”

PARTNERSHIP: “Finding 2.i. Only four of the 24 groups are clearly linked with other stakeholders and two were supported by EFI to create these linkages.”

Consider doing one!

Interactive Webinar: Sustained Exit? Prove it or Improve it! (Nov 6 2020)

Sustained Exit? Prove it or Improve it!

(reposted from Medium https://jindracekan.medium.com/sustained-exit-prove-it-or-improve-it-702ac507e2a5)

Do we exit global development projects knowing our impacts are sustained? We hope so. As Professor Bea Rogers of Tufts said after evaluating 12 projects 2 years post-closure ( https://www.fsnnetwork.org/resource/exit-strategies-study), “ Hope is Not a Strategy”, yet too often that is what projects that assume sustainability does. They/we hope. But is this good enough? For me, confirming that hope means evaluating beyond exit to ex-post, at least 2 years after donor investments end. 99% of the time, donors & development practitioners don’t return to see what lasted, what didn’t, why nor what emerged from people’s own effort. Yet we implement similar programs over and over onward, not learning lessons from the past. Sigh.

We need to evaluate what we expected to remain from our implemented projects. We also need to learn from what evaluator Bob Williams calls, “the sustainability of the idea that underpinned the results (even if the results were no longer evident)”. This is often beneath what emerged: Our projects catalyzed the local’s desire to sustain activities: taking new ways, that are locally manageable (changing how the development idea is implemented onward) or even having entirely new initiatives emerge from the participant groupings — from their own priorities, not ours. (For more on emerging impactshttps://www.betterevaluation.org/en/themes/SEIE)

Evaluation leaders talk about power, they talk about the environment. After 7 years of researching and evaluating projects ex-post evaluations, I have found there are no brilliant 100% sustained + projects nor are there any 100% abjectly scorched earth ones either. Our results are middling at best. And therein lies the rub. Projects are what donors want to give. Sometimes that overlaps with what recipient countries want, sometimes not. Most of the time the resources to sustain our multimillion-dollar, -euro, -yen, etc., investments aren’t there. We can use incentives (e.g. food aid or cash) that can bolster short-term success while we spend, but once phased out, can lead to sustainability sharply falling off as early as 2 years after we exit. It’s because while ‘development’ is about ‘our’ spending on ‘our’ programs, about short-term success while we’re implementing, rather than our equal partners’ priorities and ability to sustain it. We misuse our power. We care about ourselves far more than the people we ostensibly went there to ‘save’.

And as esteemed evaluators Andy Rowe/ Michael Quinn Patton noted, given climate change we need to question even more assumptions about how sustained and resilient our programming can be, by evaluating the natural environment on which our programming relies pre, during & implementation, at exit and ex-post closure. (More on sustained environment: https://valuingvoices.com/sustaining-sustainable-development/)

It also means we need to talk to those to whom we will eventually hand over early on to make sure we’ve built-in resilience to the climatic, economic shocks we know of so far. I recommend my colleagues Holta Trandafili and Isabella Jean’s presentations on partnering we did a couple of months ago: https://valuingvoices.com/sustainability-ready-what-it-takes-to-support-measure-lasting-change-webinar/

Finally, I have come to see that to make sustainability more likely for years to come, we must fund, design, implement, and monitor/ evaluate For Sustainability throughout the project cycle. I have come to see that folks need guidance to help support their integration of sustainability throughout, including environment & resilience, benchmarks, and more. We can learn from what ex-posts teach. Join me please, to help craft more sustained development:

Upcoming Sustained Exit Webinar: 6 Nov 2020, 14:30–17 CEST, 8:30–11 EST

“Sustained Exit? Prove it or Improve it!” Interactive webinar discussion of ex-post sustainability evaluation lessons and how to integrate into ongoing #aid programs. On Zoom, participants get resources: checklists, slides, recording, Join us to #sustain #impacts! Register, sliding scale: https://sustainedexit.eventbrite.com

Setting a higher bar: Sustained Impacts are about All of us

Setting a higher bar: Sustained Impacts are about All of us
Global development aid has a problem which may already affect impact investing as well.

It is that we think it’s really all about us (individuals, wealthy donors and INGO implementers) not all of us (you, me, and project participants, their partners and governments). It’s also about us for a short time.

 

All too often, the measurable results we in global development aid and Corporate Social Responsibility (CSR) funded projects that last 1-5 years track and report data for two reasons:

1) Donors have Compliance for grantees to meet (money spent, not lost, and results met by fixed deadlines of 1-5 years – look at some of the European Commission Contracting rules) and

2) Fund recipients and the participants they serve are accountable to ‘our’ donors and implementers who take what happened through their philanthropic grants as ‘their’ results.

Both can skew how sustainably we get to create impacts. An example of such strictures on sustainability from USAID.  As respected CGDev Elliot and Dunning researchers found in 2016 when assessing the ‘US Feed the Future Initiative: A New Approach to Food Security?‘ the $10.15 billion leveraged $20 billion from other funders for disbursement over three years (2013-16). “We are concerned that pressure to demonstrate results in the short term may undermine efforts to ensure any impact is sustainable…. Unfortunately, the pressure to show immediate results can encourage pursuit of agricultural investments unlikely to be sustained. For example, a common response to low productivity is to subsidize or facilitate access to improved inputs… it can deliver a quick payoff… however, if the subsidies become too expensive and are eliminated or reduced, fertilizer use and yields often fall…..

With so much focus on reporting early and often about the progress in implementing the initiative, there is a risk that it increases the pressure to disburse quickly and in ways that may not produce sustainable results. For example, for 2014, Feed the Future reports that nearly 7 million farmers applied “improved technologies or management practices as a result of U.S. Government assistance,” but only 1,300 received “long-term agricultural sector productivity.” Are the millions of others that are using improved inputs or management practices because of subsidies likely to have these practices sustained? And how likely are they to continue using improved practices once the project ends?”

 

3) Impact investors stick to the same two paths-to-results and add a new objective: market-competitive financial  returns. They also need to show short-term results to their investors, albeit with social, environmental and governance results like non-profits (future blog).

4) Altruists create things we want ‘beneficiaries’ (our participants) to have. For instance a plethora of apps for refugees cropped up in recent years, over 5,000 it is estimated, which can be appropriate, nor not so helpful. Much like #2 above, ‘we’re’ helping ‘them’ but again, it seems to be a ‘give a man a fish’… and my fish is cool sort of solution… but do our participants want/ need this?

 

How often is our work-for-change mostly about us/by us/ for us... when ideally it is mostly about ‘them’ (OK, given human self-interest, shouldn’t changes we want at least be about all of us?).

All too often we want to be the solution but really, our ‘grassroots’ clients who are our true customers need to generate their own solution. Best if we listen and we design for long-term sustainability together?

 

As the Brilliant Sidekick Manifesto stated in two of its ten steps:
a)I will step out of the spotlight: Sustainable solutions to poverty come from within are bottom-up, and flow from local leaders who are taking the risks of holding their politicians accountable and challenging the status quo.”

b)I will read “To Hell with Good Intentions” again and again: Politicians, celebrities and billionaire philanthropists will tell me that I can be a hero. I cannot. The poor are not powerless or waiting to be saved. Illich will check my delusions of grandeur.”

 

We have examples of where we have stepped away and participants had to fend for themselves. At Valuing Voices, we’ve done post project-exit evaluations 2-15 years afterward. What did participants value so much that they sustained it themselves (all about them, literally)? These Sustained and Emerging Impacts Evaluations (SEIE) also give us indications of Sustained ROI (Sustained Return on Investment (SusROI) is a key missing metric. As respected evaluator Ricardo Wilson-Grau said in an email, “I think calculating cost-effectiveness of an intervention’s outcomes would be a wonderful challenge for a financial officer searching for new challenges — if not a Nobel prize in economics!”)

Most of these evaluations are pretty bad news mixed with some good news about what folks could sustain after we left, couldn’t and why not. (These are the ones folks expect to have great results, otherwise they wouldn’t share them!)  While most clients are understandably interested in what of ‘theirs’ was still standing, and it was interesting disentangling where the results were attributable by implementation or design or partnership flaws or something else, what was mesmerizing was what came from ‘them’.

The key is looking beyond ‘unexpected’ results to look at emerging impacts that are about ‘them’ (aka what we didn’t expect that was a direct result of our project, e.g. spare parts were no longer available to fix the water well pump once we left or a drought rehabilitation water project that decreased violence against women), to what emerging results are attributable not to use but only to our participants and partners who took over after our projects closed.  One example is a Nepalese project ended yet the credit groups of empowered women spawned groups of support groups for battered women. Another is a child maternal health project changed how it worked as women reverted to birthing at home after NGOs left; community leaders punished both parents with incarceration in the health clinic for a week if they didn’t given birth there (wow did that work to sustain behavior change of both parents!).

Many of us at Valuing Voices are shocked that funders don’t seem that interested in this, as this is where they not only take over (viz picture, sustaining the project themselves), but they are making it theirs, not oursImagine assuming the point of development is to BE SUSTAINABLE.

Source: Community Life Competence

Our participants and national stakeholder partners are our true clients, yet… Feedback Labs tell us Americans alone gave $358 billion to charities (equivalent to the 2014 GDP of 20 countries) – in 2014 but how much of this was determined by what ‘beneficiaries’ want? Josh Woodard, a development expert, suggests a vouchers approach where our true clients, our participants, who would “purchase services from those competing organizations… [such an] approach to development would enable us all to see what services people actually value and want. And when we asked ourselves what our clients want, we would really mean the individuals in the communities we are in the business of working with and serving. Otherwise we’d be out of business pretty quickly.”

This opens the door to client feedback – imagine if participants could use social media to rate the sustained impacts on them of the projects they benefited from? A customer support expert wrote in Forbes, “Today, every customer has, or feels she has, a vote in how companies do business and treat customers. This is part of a new set of expectations among customers today that will only grow ... you can’t control product ratings, product discussions or much else in the way of reviews, except by providing the best customer experience possible and by being proactive in responding to negative trends that come to the surface in your reviews and ratings stronger.”

So how well are we working with our participants for ‘development’ to be about them?

What do you think?

Local Accountability and Transparency… During and Post Project?

 

Local Accountability and Transparency… During and Post Project?

 

Local development partners? Check. Long-term transparent and accountable investments through them as “local solutions” partners? Not so much. While President Obama and former United States Agency for International Development (USAID) Administrator Raj Shah promised up to 30% of all contracts would go to ‘local solutions’ that “promote sustainable development through high-impact partnerships and local solutions”, nowhere near that percentage became true then, much less now. While there seem to be good examples such as Haiti [1], Afghanistan is a poorer example. While most international non-profits implement projects through local sub-contractors, certainly building their capacity to manage and account for foreign taxpayer dollars spent, like this MSI in Lebanon example [2], if we extend the measure of ‘success’ beyond our project implementation, policies and programming needs to change to sustain capacity and implementation post-exit (see INTRAC report [3]).

How local partners are presented can appear as somewhat of a shell game. For while Haiti and Afghanistan have been featured by USAID, I have never seen a full inventory of partners for even a handful of the 60+ countries and regional missions that USAID works in. We hear about ‘local solutions’ and undoubtedly USAID’s ‘implementing partners’ do much good using local subcontractors. Yet are the locals winning the contracts these days? USAID posts contracts lists, for instance those who ‘won’ contracts amounting to $4.68 billion in 2016 [4]. The for-profits of Chemonics ‘won’ over $1 billion, then Tetra Tech and DAI got $800 million of contracts each [4]. These three contractors comprised 39% of all USAID obligated contract funding that year, whereas (U.S.) non-profits garnered 13% of the contracts and small and woman-owned businesses 12% and 7% [4]. Only Kenya Medical Supplies Authority, a state corporation, was listed in the top 20, winning a five-year $122 million contract for Kenya [4].  There are no equivalent sub-contractor lists, much less amounts allocated to national NGOs which would prove we are building ‘development’ ground-up.

While I am focusing on USAID, I believe this is true of most bilateral and multilateral donors. For USAID, caveats abound regarding their ability to accomplish local and sustained ‘development’. A 2015 Congressional Research Paper about their Background, Operations and Issues, cites “multiple challenges in the course of fulfilling its mission” [5], including:

  • Local Solutions. Providing assistance to local entities incurs the risk of loss of taxpayer dollars. Efforts to mitigate risk generally require more personnel and consequent funding to monitor local entities and build their capacities [5]
  • Sustainability. ‘Country ownership’ and domestic resource mobilization efforts are two ways the agency has sought to address sustainability, but a clear path to sustainability remains a work in progress [5] …

[Yet] the agency argues that investments are best sustained in the long-term if development is locally owned, locally led, and locally resourced.”

 

For more accurate accountability and transparency for bilateral, multilateral, pro-profit and non-profit implementers, we must look within data underlying the ‘development’ allocations abroad. For instance, the US government’s country-level foreign appropriations overall budget for 2017 (see Table3a) shows that $36 billion funded a variety of branches of the US government’s ‘development’, it would be instructive to see what the amounts of the funders’ award contacts which would be broken down into: what % went to implementers, what % went to national governments or local contractors, and what % was directly used for our participants [6]. Maybe this is a new aspect the industry-standard Charity Navigator can add to its existing Accountability and Transparency criteria. On my repeated wish list for them is to show evidence the nonprofit is systematically doing and learning from post project sustained impacts evaluations. I first asked this 5 years ago 🙂

While I am scratching the surface, at least one private sector Corporate Social Responsibility company seem to have more transparent systems. This balance sheet from Abengoa, a sustainable energy technology company, could be updated with such a breakdown.

[7]

Their website also talks about its 25-year CSR investments (which is an enviable timespan, for most donors have 1-5 year projects) [7]. “Abengoa believes that the good relationship it has with local communities, as well as respect and development in the areas where it operates, reaps benefits, referring to this method as “social licence to operate” [7]. Abengoa’s social engagement aims to further the social and cultural development of the communities where they operate. From 2014 to 2016, the company reported its social performance in line with the criteria proposed by the London Benchmarking Group (LBG) methodology. This model defines a method to measure, manage, assess and disclose contributions, achievements and impacts of the company’s social engagement with the community” [7].

Their website also describes current events. “A flagship initiative of the company is the PE&C (People, Education and Communities. Committed to Development) programme.… is now present in nine countries (Argentina, Peru, Brazil, India, Mexico, Chile, Spain, Sri Lanka and Morocco), [but] currently, the complex situation that the company is undergoing and the severe limitation of financial resources in recent months has meant the gradual and temporary reduction of the contributions made to social projects in the different regions. As part of the restructuring plan agreed with creditors and in order to limit the social engagement items based on the resources available in the different business units, each company has assessed its capacity to fund social development projects, maintaining, in some cases, their commitment to certain local social projects. In an effort to avoid the negative impact on these communities and disadvantaged groups, the company has worked hard to find partners and collaborators who could provide continuity to these projects until Abengoa can recover a solid economic position that allows it to continue working and giving support to them” [7] (which their Press Room tells us they have in 2018).

Sharing “achievements and impacts of the company’s social engagement with the community”, “find[ing] partners and collaborators who could provide continuity to these projects” is not often done in bureaucratically time-fixed global development [7]. For too rarely do the fixed timelines and budgets, inflexible metrics and demanding deliverables enable true partnerships. Save the Children’s 2008 brief on aid modalities for country ownership includes a vital point, which is willingness. “The United States lags behind other donors in its willingness to use all the aid mechanisms that would build capacity, such as channeling aid through host countriessystems”, including where only “1 percent [of aid] was passed through projects directly implemented by the host government” [8]. At least US funds were aligned with national government priorities in Bangladesh, Ethiopia and Malawi and in Liberia the US partnered with UK’s DFID for community infrastructure. Nonetheless, even aid to governments must be as locally transparent as possible for true accountability. “Local NGOs are a key actor in holding host governments accountable for the delivering meaningful results, and should increasingly be an important link between government and community through communications and provision of services” [8] … So why aren’t we funding more? We dont know, dont much data how well we are, but unless we start with accountability to the country nationals we are ostensiblydeveloping’, sustained success will not ensue.

 

The scant number of post project sustainability evaluations have shown how rarely our international donor funded partners return to partners and participant communities to see what they could self-sustain after our projects ended.  So much for accountability to our true clients! Public and private sector needs to turn away from being data extractors aiming at shortterm results, and rather turn to being led by sustained partnerspriorities and myriad voices. Private sector companies may have lessons to teach, for would they stay afloat if its investors did not learn how well their product worked by not returning to ask after the sale assess client satisfaction?

We don’t have a moment to waste.

Thoughts? Questions? Look forward to your comments.

PS – There are surely 500 sources I didn’t find in time to include, including this blog regarding Cambodia and aid, huge numbers of organizations focused on capacity building and also thanks to Abu Ala Mahmudul Hasan for a Pelican online discussion that spurred this. We hope to create a podcast this spring, so stay tuned…

 

 

Sources:

[1] USAID Haiti. (2017, March). Local Solutions: Building Up Haitian Organizations (Fact Sheet). Retrieved from https://www.usaid.gov/sites/default/files/documents/1862/FINAL_Local_Solutions_Fact_Sheet_March_2017.pdf

[2] MSI, A Tetra Tech Company. (2013, December 17). USAID/Lebanon BALADI CAP Overview w/ Dr. Fares El Zein. Retrieved from https://www.youtube.com/watch?v=TzHvbDuekLI

[3] Hayman, R., & Lewis, S. (2017). INTRAC’s Experience of Working with International NGOs on Aid Withdrawal and Exit Strategies from 2011 to 2016. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 29, 361-372. Retrieved from https://link.springer.com/article/10.1007/s11266-017-9901-x

[4] Devex. (n.d.). Top USAID Contractors for 2016. Retrieved from https://pages.devex.com/rs/685-KBL-765/images/Devex_reports_USAID_Top_Contractors_in_2016.pdf?aliId=2107099003#:~:text=Chemonics%20reclaimed%20its%20position%20as,the%20contract%20funding%20from%202015.

[5] Congressional Research Service. (2015, July 21). U.S. Agency for International Development (USAID): Background, Operations, and Issues. Retrieved from https://crsreports.congress.gov/product/pdf/R/R44117

[6] US Department of State. Congressional Budget Justification, Foreign Assistance: Summary Tables FY17. (n.d.). Retrieved from https://www.usaid.gov/sites/default/files/252735.pdf

[7] Abengoa. (2016). Annual Report 2016. Retrieved from http://www.abengoa.com/web/en/accionistas_y_gobierno_corporativo/informes_anuales/2016/

[8] Save the Children. (2010, May 27). Aid Modalities for Country Ownership. Retrieved from https://www.savethechildren.org/content/dam/usa/reports/advocacy/aid-modalities-for-ownership-2010.pdf

 

Living in a Well-meaning Lie: Valuing all Voices? – The Solutions Journal

Living in a Well-meaning Lie: Valuing all Voices?

By Jindra Cekan


Solar panels in Cap Haitien, Haiti.

Poor villagers like Edith, Aminata, Rituu, and Juan don’t appear much on the nightly news.  You might never know they exist unless you stop and read your mail from some charity asking you to help them.  On the brochures, they can look scared or sad; maybe surrounded by their thin children, with a parched land or dying animals behind them. Our foreign aid programs should be helping them, but are they?

 

I have met these people while they are working in their fields, growing corn and peas, millet and coffee, raising their chickens and goats. I have talked to them outside of health centers where they have brought their babies to be vaccinated or their parents for medical care.  I got to know them when I interviewed them under the big tree in the middle of their village, or in empty school rooms, asking them what they need from us, and how we can design foreign aid projects to better help them.

 

I have worked in international development as a technical expert in project design and monitoring and evaluation for international non-profits such as Catholic Relief Services, Save the Children, the Red Cross, and many others including the Bill and Melinda Gates Foundation and the US Agency for International Development (USAID). I estimate that I have designed and evaluated over 200 projects in 28 countries over the last 29 years. I have felt lucky to do this work, and foreign aid does achieve some very good work while we are there: helping farmers to farm better, or helping men and women to care for their family’s health, wealth, and future with new knowledge, tools, and items they need for daily living. My colleagues do wonderful work as well, in hard conditions, within countries with few resources, and for donors with unrealistic expectations of how much can be done well in short timeframes.  In 2010, USAID stated that they would aim for 30 percent of funding to be spent by national partners under USAID Forward.1 This is an excellent step toward the country-led development that the Paris Accords promised, yet as of 2016, there is no list of local partners, other than a handful of examples.2 The only ‘country partners’ list posted to the USAID website includes 80 organizations doing programming via USAID in Afghanistan alone, 55 of which are American firms, four US agencies, nine Afghan government-affiliated organizations, six foreign governments, six UN agencies, and two MENA firms.3 Not quite the national civil-society-NGO partners we envisioned in 2010. Under the new U.S. administration, these are likely to shrink even more as the 0.5 percent of our GNP we allocate to foreign aid is redirected inward—that much more reason to make it as sustained as possible.  European aid as well as other rising world nations need this approach just as much.

Large parts of international aid system remain broken. We design too many projects outside of the countries themselves. We have fixed funding and leave in pre-set times rather than when participants are actually ready to take over. We ‘handover’ without partnering throughout the whole project so that partners can determine what they are able to sustain. Even worse, we leave and do not look back to learn from our Ediths and Juans after our projects have closed.  Sometimes, we disparage their knowledge, and at other times we don’t make enough time to ask but wish we did. Mostly, our aid industry is designed around measuring success while we intervene, and then abruptly leave because funding ended. Yet development is, as international evaluator Ian Davies says, “A process, not a result.”

 

Our policies say we are doing “sustainable development”, that we are helping our ‘beneficiaries’ (really our partners and participants) feed themselves over the long-term, that our projects are almost all successful, and that all we need to do is to scale up the great projects out there.  But the numbers prove we are not, in fact, achieving sustainable development. Nine times out of ten, we rarely go back to talk to our participants and partners after our project end, and we move on.

 

The numbers are staggering:

  • Of the US$5 trillion dollars of international foreign aid spent since 1945, we have evaluated the long-term sustainability far less than one percent of the time.
  • Since 2000, for example, USAID and the Millennium Challenge Corporation have only done three such evaluations apiece, yet they spent well over US$300 billion.
  • The EU evaluated only a few dozen of its projects and programs, in spite of spending US$1.5 trillion in the last 15 years. The United Nations Development Program may do up to six a year, and the World Bank more, but how often do any of them talk to project participants and design anew based on what we learned that succeeded and failed?
  • The Japan International Cooperation Agency, and to some degree EU bilateral countries (through the Organization for Economic Cooperation and Development), have evaluated the sustainability of over 300 projects.
  • Despite this, tens of thousands of new projects are launched every year.
Cekan 2
Satellites atop homes in a slum in Tigray, Ethopia.

 

This is why I founded Valuing Voices—to analyze what little we know to make development better.4 Not to destroy international development, but to change how we fund, design, and implement it. We need to design for sustainability of the activities by the country nationals themselves, rather than designing for results we can show to get more funding. We also must jointly implement, monitor, and evaluate our projects so countries can continue after we leave. Smaller organizations can do even simple activities, designing projects based on what the participants feel they can self-sustain, and partnering with those who will take over while they are still there.

 

Having spread the word for the last three years, to mostly little response, I now turn to you, readers.  Our analysis – and a wonderful 2012 book, Time to Listen by Mary Anderson and Dayna Brown, shows that sometimes when our projects partner with country nationals, their people become – and stay – better off.5They want to be engaged, yet our very structure of delivering aid prevents this. Often we are not there long enough to make a lasting difference, or we invest scarce time on untested innovations that work in some places but don’t in others. Even worse, sometimes we design activities so badly that villages are left with irrelevant technology and trainings, wasted funding, and lost hopes. At other times, there are successes as well, but not returning robs us of the chance to replicate those. We do ‘impact evaluations,’ but only on successes during project implementation, and not on what people can self-sustain after we leave. Our vision is so limited. Our well-meaning self-interest blinds us.

 

Across the board, our development projects make one massive and incorrect assumption that once we ‘handover’ the project, the local government, community, and households have the means to sustain our multi-million dollar investments.6 We assume that technical knowledge will still be locally available to the villagers, that inputs like seeds and tools, data and vaccines will be accessible both physically and financially, that the government staff have the means to get to villages or that new NGOs and donors will appear to fill the gaps. The Huffington Post has stated, “as long-term projects and action-plans are established, more investment must go into financing locally designed solutions and projects that ensures ownership is placed back into local communities.”7 While more project have begun using feedback loops of listening to participants during implementation, virtually all good work stops when project funding stops.

 

Don’t we want development to be sustained after resources leave, and the opinions of these aid recipients to be heard? Don’t we want the next project to address the needs better?  Don’t we, as taxpayers, want to demand that agencies using our tax dollars learn what is really sustainable and what is not?  And shouldn’t we demand that all projects costing more than US$1 million over the past 10 years be examined now for lessons learned by sector (agriculture, health, credit, education, etc) and region? Shouldn’t post-project sustainability evaluations be included in all new projects?  Don’t our participants and partners deserve the dignified futures they hope for, our creating channels for their voices that enable them to evaluate us and teach us how we can help them to be successful?

 

In fact, a radical Foreign Aid Transparency Act was just passed in the U.S. in June 2016.8 The bill calls for the President, within 18 months of enactment, to “set forth guidelines…for the establishment of measurable goals, performance metrics, and monitoring and evaluation plans that can be applied with reasonable consistency to covered United States foreign assistance.” These include ‘ex-post’ (sustainability) evaluations, and “can have enormous value when it comes to making programming and budgeting decisions.”  Yet while there is a call for guidance to be developed, no funding came along with this bill.  Without the funds to make this happen, this may be more ‘window dressing’ for sustainable development than excellent policy.

Cekan 3
A small enterprise in Cap Haitien, Haiti.

 

There is some hope coming from the corporate sector. While impact investors are often more focused on return from emerging economies than fostering sustainable development, corporate social responsibility is building bridges in lovely ways.  Tsikululu Social Investment of South Africa has thought about what advice to give to the companies they advise on such investments, as well as exiting from them.9

 

We argue that our budgeting needs a basic business metric: Return on Investment. In a time of huge demands on our resources worldwide from refugee flows, terrorism and climate change, we currently do little or no analysis of:

  • How much actual investment: What percent of allocated funds went to the activities that benefitted the partners and participants themselves, rather than being used as overhead for operations?
  • How much return: What is the value of what remains used 3-10 years after we leave? What was the value-added that communities and other funders (including the national governments themselves) who followed catalyzed based on our earlier investments? What were unexpected new results that emerged?

 

We envision a beautiful future, one where Edith, Aminata, Rituu, and Juan and their local partners are at the center of development.  We imagine a world in which we listen to what people in need can sustain for themselves. Through these approaches data is shared widely on what has worked best and why; aid projects invest in country-systems and staff that boost their ability to self-sustain; and only sustainable projects are designed and funded that foster country-led development. The global adoption of the Sustainable Development Goals finally puts the focus on all we can do to foster sustainability of our work.

 

By valuing voices and focusing on sustainable solutions for excellent impacts, this will promote truly sustainable development from our aid organizations, government and non-governmental alike. We have much to learn, and there is not a moment to waste before we start Valuing Voices of those we serve and partner with on country-led development.

 

References

  1. USAID Forward. USAID [online] (2017). https://www.usaid.gov/usaidforward.
  2. In-country Partners. USAID [online] (2016). https://www.usaid.gov/partnership-opportunities/in-country-partners.
  3. Implementing Partners. USAID [online] (2017). https://www.usaid.gov/afghanistan/implementing-partners.
  4. Valuing Voices [online]. https://valuingvoices.com/.
  5. Jacobs, A. Time to Listen by Dayna Brown and Mary B Anderson. NGO Performance [online] (December 4, 2012). https://ngoperformance.org/2012/12/04/time-to-listen-by-dayna-brown-and-….
  6. Cekan, J. What happens after the project ends? Lessons about funding, assumptions and fears (Part 3). Valuing Voices [online] (February 29, 2016). https://valuingvoices.com/what-happens-after-the-project-ends-lessons-abo….
  7. Zuabi, V. Investing in Locally Designed Solutions for Syria and the Middle East. The Huffington Post [online] (May 24, 2016). http://www.huffingtonpost.com/vanessa-zuabi/investing-in-locally-desi_b_….
  8. Ingram, G, Miles, C & Veillette, C. The Foreign Aid Transparency and Accountability Act is Law! Now What? Modernizing Foreign Assistance Network [online] (August 2, 2016). http://modernizeaid.net/2016/08/foreign-aid-transparency-accountability-….
  9. Cekan, J. Towards responsible donor exiting strategies and practices: Reblog from Tshikululu. Valuing Voices [online] (October 8, 2016). https://valuingvoices.com/towards-responsible-donor-exiting-strategies-an….

Source: Living in a Well-meaning Lie: Valuing all Voices? – The Solutions Journal

How ‘new’ are our projects… and who is aiming at the right outcomes?

 

How ‘new’ are our projects… and who is aiming at the right outcomes?

 

Valuing Voices exciting news is we have received research grant funding from the esteemed evaluator, Michael Scriven’s Faster Forward Fund. We’re looking into the value-added of (ex-post) Sustained and Emerging Impacts Evaluations (SEIEs) and we are doing the research now. We will be documenting methods used and discuss how best to evaluate such sustained impacts after project close-out. Very exciting stuff in this staggeringly ‘new’ field of evaluation.

During this research, a senior international development evaluation expert told us that they can’t return to evaluate now-closed projects because they aren’t the same projects anymore (after closeout) and we are no longer responsible for the results. That took my breath away.

All new projects come from old projects… we recycle old project design most of the time, occasionally making substantive changes in targeting or design but much of how we design and implement remains the same. And while we thoroughly evaluate them during implementation, learning ex-post is a key missing link which all projects in the future can benefit from as we do similar interventions and track similar outcomes year after year but we rarely know which ones were sustained or emerged anew. There absolutely are aspects that get adapted but there are only so many ways to heal the sick, improve crop growth, save money, learn to read and so on, and there is a world we need to learn about what enabled some to be sustained and even morph into new results!

This excellent article, Do NGOs (non-governmental organization) help?, notes that “due to donor pressure [NGOs] are increasingly forced to respond with a discrete project with x number of deliverable outcomes” [1]. It goes on to cite D. Sriskandarajah, the secretary-general of Civicus, a global network of civil society organizations and activists, wrote: ‘We have become a part of the problem rather than the solution… Since demonstrating bang for your buck has become all-important, we divide our work into neat projects, taking on only those endeavours that can produce easily quantifiable outcomes. Reliant on funding to service our own sizeable organizations, we avoid approaches or issues that might threaten our brand or upset our donors. We trade in incremental change’” [1].

We also settle for results while we control them, and don’t ask unpopular questions about who is to sustain these results, with what resources, and for goodness sake, why sustained impact was not funded, designed, implemented and monitored/ evaluated from the very onset in our rush to measurable results?

 

DontSettleForLess

 

As this great NGO article by Dinyar Godrej goes on to say, “most media scrutiny of NGO accountability is of how they use funds, their accountability to donors. But what of their accountability towards the recipients of their interventions” [1]? They have no lobbyists to persuade our funders they would like this but nto that, and often such lobbying for their needs falls to the very NGOs that have won these large contracts and tasked with implementing a dizzying array of mandatory input, output, outcome and some impact indicators. We do care deeply about results! US State Department/ USAID has a “Standard Foreign Assistance Master Indicator List” of 2,300 lines in an excel spreadsheet [2]. (There are more indicators still– custom and cross-cutting indicators, the mind boggles).

Wow. But are we asking the right questions? Are we asking what was sustained after all this hard work was done and ended? Rarely. Who should be?! “It is perhaps unrealistic to expect such large structural changes to be delivered by NGOs when governments don’t tackle them either.”

For the rub is this. When we take development over from national governments, largely do not involve country nationals in the funding, design and M&E of projects, then how sustained can these projects still be after we go? Millions are invested, then disappear… Last year, at local debrief at the end of one SEIE Valuing Voices did, the state of affairs became crystal clear when a government official asked us “Can you ever find some funds to fund us to do our own independent evaluations? Even if it is not the projects that they did themselves? We would be happy to get that support…”

When are we no longer responsible for doing great, sustained work? Valuing Voices will let you know what we found regarding the best ways to do SEIEs more. Stay tuned.

What do you think?

P.S. This blog topic prompted me to look for statistics on the number or percentage of funded projects that were renewed. Nothing.  Does anyone know how many or what % of projects were extended/ funded again after showing good results? (Often this happens in the form that a successful project in one area of the country gets either funded again or repeated elsewhere in the country or in the world, as have two of our own SEIEs, Niger and Ethiopia). For that matter, what made them so excellent to be replicated? What can we learn?

 

 

Sources:

[1] Godrej, D. (2014, December 1). NGOs – Do They Help? Retrieved from https://newint.org/features/2014/12/01/ngos-keynote/

[2] US Department of State. (n.d.). Standard Foreign Assistance Indicators. Retrieved 2017, from https://web.archive.org/web/20170404072145/https://www.state.gov/f/indicators/index.htm