Wishing for ex-post evaluation Christmas Lights rather than Needles in Haystacks

 

Wishing for Ex-Post Evaluation Christmas Lights
Rather than Needles in Haystacks

 

This is what the life of most ex-post evaluation researchers looks like, mostly without the counting congratulator:

 

I recently spent three days looking for ex-post evaluations for a client across nearly a dozen organizations. I was hard-pressed to find 16 actual ones. Sorting through ‘impact evaluations’ that were done in the middle of implementation does not tell us anything about what was sustained after we left, nor do delayed final evaluations that happen to be done after closure. While these (rightly) focus on cost-effectiveness, relevance, and efficiency, measures of sustained impact are projections, not actual measures of what outcomes and impacts stood the test of time. I weeded out some desk studies that did not return to ask anyone who participated. Others titled ‘ex-post’ were barely midterms (I can only gather they misconstrued ‘ex-post’ as after-starting implementation?), and a few more reports only recommended doing ex-post evaluation after this final evaluation. For more lessons on how random and misconstrued ex-posts can be, see Valuing Voices’ research for Scriven. None of these 16 actual ex-posts even told us anything about what emerged (as we look at during Sustained and Emerging Impacts Evaluations) from local efforts in the years after assistance ended [1].

This is what I wish my ex-post haystack would look like, bountiful treasures of numerous ex-post-project evaluations, as numerous as these Christmas lights here in Tabor, Czech Republic.

 

If we had more ex-posts to learn from, we could learn from what lasted. What could locals sustain? Why? Why not? How can we do better next time? We could compare across sectors and countries, and we could see what conditions and processes during implementation supported sustainability -and, importantly – why some failed, so we don’t repeat those mistakes.

We could move from our current orange slices that end at closure to green sustainability of the project cycle:

I will be adding the ones I found to our Catalysts list soon, but when my client asked me who held databases of ex-post evaluations, I had to say only Valuing Voices and Japan’s JICA (since 1993, who even differentiates the ex-posts between Technical Grants and ODA Loans). This is not to say some cannot be found by trawling the OECD or the World Bank, but this is Needle-in-Haystack work again, and so there are only two databases to learn from. Isn’t that shocking?

Now JICA has really upped the illumination ante, so to speak: They are now doing what they are calling ‘JICA’s Ex-post Monitoring’, which was like Christmas come early [2]! Returning to learn at least 7 years after the ex-post, which was 1-3 years after closure, such as in this case of ex-post monitoring and learning from 10 projects (2007). They have done ex-post monitoring for a total of 91 cases, evaluating the sustained impacts of results, see if JICA’s recommendations to their partners had been implemented, how they had adapted to changes over a decade post-closure, and find learning for new programming. “Ex-post monitoring is undertaken 7 years after a project was completed in principle in order to determine whether or not the expected effects and impacts continue to be generated, to check that there are no sustainability-related problems with the technical capacities, systems, and finances of the executing agency nor with the operation and management of developed facilities, etc., and to ascertain what action has been taken vis-a-vis the lessons learned and recommendations gleaned during the ex-post evaluation.” While it was unclear why these specific projects were selected, it is amazing they are doing 5-10 per year.

They are my ex-post gods/ goddesses, and I fawned over two JICA evaluators at the last European Evaluation Society Conference. Why do I fawn? JICA lists 2,273 results under ex-post evaluations of Technical Cooperation, Grant Aid, ODA loans! They are literally the only organization I know whose searched reports are actually ‘ex-post’.

What we can learn from returning again is illustrated by one of JICA’s water project loans in RSA, which ended in 2003, had an ex-post in 2006, followed by monitoring of sustainability in 2013 [3]. While the report included issues of data access and evaluators expressed caution in attributing causation of positive changes to the project, but it not only continued functioning, the government of South Africa (RSA) solved barriers found at the ex-post:

  • “Data for the supply and demand of water pertaining to the Kwandebele region could not be obtained. However, considering the calculation from the water-supplied population and supplied volume and the result from the DWAF interview, water shortage could not be detected in the four municipalities studied by this project…” [3]
  • “The ex-post evaluation indicated that the four components were not in the state to be operated and managed effectively. Currently, the components are operated and managed effectively and are operating under good condition [and] concerning sustainability, improvement can be seen from the time of ex-post evaluation. Shortage of employees and insufficient technical knowledge has been resolved…” [3]
  • “Compared to the time of ex-post evaluation, improvement was seen in the under-five mortality and life expectancy. However, since the components implemented by this project are limited in comparison with the scope of the project, it is impossible to present a clear causal relationship” [3].

In another, from Indonesia’s air quality testing labs which involved capacity building and equipment maintenance 6 years after the ex-post, they mostly found training and use continued despite organizational changes and maintenance challenges: [4]

  • “After the ex-post evaluation, many of the target laboratories changed their affiliation from the Ministry of Public Works (MOPW) and MOH to provincial governments. While the relocation of equipment has been carried out in a handful of provinces, in other provinces, equipment is still located at the laboratories where it was originally installed, and these laboratories still have the right of use” [4]
  • In spite of some irregularities,”As the Ministry of Environment (MOE) still has ownership of the equipment, some laboratories have inappropriate audit results that show the allocation of O&M budget to equipment which is not included in their accounting…” [4]
  • “Out of 20 laboratories where the questionnaire survey confirmed that equipment still remained, 15 laboratories replied that spare parts for equipment are still available but are difficult to obtain…It takes several months to one year to obtain spare parts, occasionally out of Indonesia, even if a repair service is available” [4].

In this case, there were lessons learned for JICA and Indonesia’s Ministry of the Environment programs about ownership and the right use of the equipment and retiring obsolete equipment. Talk about a commitment to learning from the ongoing success or failure of one’s projects!

 

As you have read here on Valuing Voices for more than 6 years unless we include post-project sustainability that asks our participants and partners how sustained their lives and livelihoods could be and even resilient to shocks like political or climate change, we cannot say we are doing Sustainable Development. We need such lessons about what could be sustained and why.

We can prepare better to foster sustainability. In the coming months, we are working on checklists to consider during funding, design, implementation, M&E pre-and post-exit to foster sustainability. Will keep you posted, but as World Vision also found: “Measuring sustainability through ex-posts requires setting clear benchmarks to measure success prior to program closure, including timelines for expected sustainment.”

And as my gift to you this Holiday Season, let me share WV’s Learning Brief about Sustainability, with wise and provocative questions to ponder about dynamic systems, benchmarking, continuous learning, attribution, and managing expectations [5]. World Vision shares how infrastructure and community groups and social cohesion fared well, yet lessons circled back to the need for JICA-like ‘monitoring’ and mirror rich ex-post lessons from FFP/Tufts (Rogers, Coates) and Hiller et al. that explains why we do ex-posts at all: “Project impact at the time of exit does not consistently predict sustainability“ [6].

 

Now my gift: a few big lessons from the 6 years of researching sustainability across the development spectrum.  I have found no evaluations that were only positive OR ONLY NEGATIVE. Most results trended downwards, a few held steady, and all were mixed. We cannot assume the sustainability of results at closure nor optimistic projections, as we’ve seen in the climate arena.

Please consider:

  • Designing with our participants and partners so what we do,
  • Implementing with partners far longer to make sure things still work,
  • Adapting exit based on benchmarks to see how well the resources, partnerships, capacities, and ownership have been transferred,
  • Using control or comparison groups to make sure ‘success’ was due to you and being careful about attributing results to your projects while considering how you contributed to a larger whole of ongoing country progress or stagnation,
  • Being willing to jettison what is unlikely to be sustained and learn from what we designed and implemented poorly (due to our design, their implementation, and external conditions),
  • Given climate change, learning fast, adaptively, and revising fast given changing conditions,
  • Without knowing what has been sustained, we cannot replicate nor scale up,
  • Sharing lessons with your leaders – for people’s lives depend on our work,
  • Learning from what emerged as our participants and partners refashioned implementation in new ways could sustain it (without the millions we brought),
  • Refocusing ‘success’ from how much we have spent to how much was sustained.

 

Please make our next Christmas merry. Do MANY ex-post evaluations, Learn TONS, Share WIDELY WHAT WORKED AND FAILED TO WORK (you will be praised!), and let’s CHANGE HOW WE DO SUSTAINABLE DEVELOPMENT.

 

May 2020 bring health, happiness, and to all of us a more sustainable world!

 

 

Sources:

[1] Cekan, J., Zivetz, L., & Rogers, P. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE

[2] JICA. (n.d.). Ex-post Monitoring. Retrieved December, 2019, from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/index.html

[3] Matsuyama, K. (2012). Ex-Post Monitoring of Japanese ODA Loan Project: South Africa, Kwandebele Region Water Augmentation Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/2012_full_03.pdf

[4] Kobayashi, N. (2009, August). Ex-post Monitoring of Completed ODA Loan Project: Indonesia, The Bepedal Regional Monitoring Capacity Development Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/indonesia2008_01.pdf

[5] Trandafili, H. (2019). Learning Brief: What does sustainability look like post-program? Retrieved from https://valuingvoices.com/wp-content/uploads/2019/12/Sustainability-Learning-Brief_final_WV-icons.pdf

[6] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fsnnetwork.org/ffp-sustainability-and-exit-strategies-study-synthesis-report

 

Learning from a river of ex-post project evaluations, tools and guidance… Thanks USAID!

Learning from a river of ex-post project evaluations and tools… Thanks USAID!

Dear ex-post aficionados. It’s raining ex-post project evaluations. Here’s hoping learning from such evaluations in water/ sanitation, maternal/child health and even capacity building/ peacekeeping, and their number increases!

 

1. WATER/ SANITATION & HYGIENE:

USAID has a series of six ex-post evaluations of the water/ sanitation and hygiene sectors since 2017! What is exciting is that they are also looking to the future. These evaluations will “provide insight into what happens after an activity ends, and how to mitigate challenges in future programming, potentially. The series will inform USAID’s WASH activity design and implementation and contribute to a larger sector discussion on achieving sustainability.”

The E3 water division (Water CKM ) took sustainability on as their strategy and have made great strides these last two years. They have done five ex-post project evaluations, cited below, and MSI has completed one more wat/san/ hygiene ex-post evaluations, specifically:

Madagascar Rural Access to New Opportunities for Health and Prosperity (RANO-HP) – Published June 2017
The first evaluation in the series explores the sustainability of the sanitation and hygiene components of the RANO-HP activity, implemented in 26 communes from 2009–2013.

Indonesia Environmental Services Program (ESP) – Published August 2017
The second evaluation in the series examines the sustainability of water utility capacity building, microcredit, and financial outcomes associated with the ESP activity, which was implemented from 2004–2010.

Ethiopia Millennium Water Alliance (MWA-EP) – Published May 2018
The third evaluation in the series examines the long-term sustainability of outcomes related to rural water point construction, rehabilitation, and management, as well as participatory sanitation and hygiene education and construction related to the MWA-EP activity, implemented in 24 rural districts between 2004–2009.

Financial Institutions Reform and Expansion–Debt and Infrastructure (FIRE-D) – Published September 2018
This evaluation is the fourth in the series. It examines how urban water and sanitation services in India have changed since FIRE-D closed and to what extent policies, practices, and financing mechanisms introduced through FIRE-D have been sustained.

Millennium Water and Sanitation Program in Senegal (PEPAM/USAID) – Published July 2019
The fifth ex-post evaluation in the series looks at the PEPAM project (Programme d’Eau Potable et d’Assainissement du Millénaire au Sénégal), implemented from 2009–2014 to improve sustainable access to WASH in four regions of Senegal.

 

USAID-funded by MSI: USAID/Ghana’s Water Access, Sanitation, and Hygiene for Urban Poor (WASH-UP)– published Nov 2018

Also USAID and Rotary International developed a WASH Sustainability Index Tool, “to assess a WASH activity’s likelihood to be sustainable according to the following factors: availability of finance for sanitation; local capacity for construction and maintenance of latrines; the influence of social norms; and governance.” This is similar to what we learned from USAID/ FFP/ Tufts/ FHI360 12 ex-posts that resources, capacities, motivation and linkages (aka partnerships, including governance) are vital to sustaining outcomes and impacts.

 

It will be interesting to see whether they examine the other ex-posts for excellent lessons, as they have the Senegalese evaluation:

  • “Whether or not to subsidize sanitation access …Based on this evaluation’s findings and exploration of the literature, subsidies can help improve the quality of household latrines, but increasing use of those latrines remains a challenge.
  • In contrast, CLTS (a nonsubsidized approach) is often credited with increasing use of unimproved latrines, but serious questions linger about quality and long-term sustainability of the latrines built after CLTS triggering, particularly as it relates to moving up the sanitation ladder. This evaluation… provides the opportunity to examine the potential value of a hybrid approach….
  • The handwashing results suggest that low-cost, low-quality handwashing stations such as tippy taps do not lead to sustained behavior change. It may be worth considering hygiene investments that reduce the behavior change burden on targeted beneficiaries.

2. MATERNAL/ CHILD HEALTH & NUTRITION:

 “Sustainability of a Community-Based CHOICE Program to Improve the Health and Nutrition Status of Mothers and Infants in Indonesia,” The report focused on whether the USAID-funded CHOICE program had left sustainable impacts: improving the health and nutrition status of children under the age of five, as well as the health status of pregnant and lactating women and mothers or caretakers of young children in the Pandeglang District of Indonesia. “After examining the data collected from the PSS, the researchers found that there were significant improvements in many indicators—such as births attended by skilled personnel, the treatment of diarrhea, and the nutritional quality of food fed to infants—in the six years after the CHOICE program ended. However, despite these improvements, the researchers found no significant statistical differences between villages that received the CHOICE program interventions and comparison villages, which did not. This speaks to using such a comparison methodology to focus on actual contribution and rule out the “rising tide lifts all boats” phenomenon.

 

3. CAPACITY DEVELOPMENT & PEACEBUILDING:

USAID’s Regional Office in Thailand evaluated its capacity building and peacebuilding program 1.5 years ex-post.  While civil society was strengthened and there were inroads made on peacebuilding, “many interventions initiated during Sapan did not continue post-Sapan, although some did remain. For example, “stakeholders cite evidence of continuing to use some governance tools in local governance related to service delivery [although] because of limited financial resources after Sapan ended, they had to change some of their interventions and reduce the range of people they could include. There are lessons for whose capacities are built, two-way feedback loops with local partners, using local organizations such as universities to sustain training, planning sufficient time for partners to internalize training lessons, etc.

 

4. USAID FUNDED GUIDANCE:

‘Impact Evaluations’ have a new focus on long-term impact, rather than effectiveness during implementation (which was at least the original intent of impact evaluation in the 1980s)! In September 2018, USAID and Notre Dame issued a Guide for Planning Long-term Impact Evaluations as part of the Utilizing the Expertise of the ERIE Program Consortium. The guide covers the difference between traditional impact evaluation designs and data collection methods and how to apply them to long-term impact evaluations (LTIE). It also shares examples across a range of sectors, including later evaluating past impact evaluations, which ended before final evaluation.

Finally, in new 2018 USAID guidance, ex-post evaluation is clarified as the source of the sustainability of services and benefits. USAID clarifies that “questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation, a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be, and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project’s services and benefits were sustained.”

Such richness that we can learn from. Keep the momentum going on the 99% of all global projects yet unevaluated ex-post, and change how we fund, design, implement, monitor and evaluate global development projects!

Implementing, Scaling and Planning for Aid Exit and Sustainability

Reposted from: https://medium.com/@jindracekan/implementing-scaling-and-planning-for-aid-exit-and-sustainability-b1b92e70fb36?postPublishedType=initial

Rarely do funders return to evaluate (ex-post) what lasts after aid projects end, but when they do, we can find myriad pleasures:

1) sustainability of activities we launched and nurtured together and hoped would last as is, even 15 years later, or

2) new ways local participants or partners made old activities last which we would have never imagined, or

3) wholly new activities or collaborations that emerged which we could not have foreseen but which meet the evolving needs of participating country nationals.

All three elicit two questions: a) ‘what did we (funders, designers, implementers, evaluators) do right during design, implementation, and exit?’ and b) ‘what did they do so well after we left?’ There are rich answers for this, which involves how we co-funded, co-designed, co-implemented, and co-evaluated all along the program cycle, and how we exited.

Did we make enough time and measurement to foster sustainability, as we phased-down and phased-over an array of activities, alongside those remaining (white paper forthcoming)? Did we abruptly phase-out leaving partners and participants at a loss? Sharing power over all these decisions will influence what lasts.


There is an amazing breadth of local, ongoing resources, skills & capacities, linkages, motivation (thanks to Tufts FHI360’s work for USAID’s Food For Peace) that we can explore and learn from. There are local innovations and an array of unplanned collaborations (e.g., funding for health staff (Niger), training in small enterprise from the national government (Bangladesh), or private sector markets (Ethiopia) that can be accessed when partnerships are transparent and created one or more years pre-exit to collaborate on post-exit.

Ideally, we design and implement for exit from the onset. When we jointly set the timeframe and jointly assess risks to sustainability and adaptively manage exit, rather than exit based on pre-set timeframes, all sides win, with partners and participants able to foster sustainability. As USAID/ GIIN wrote about Responsible Exits for Impact Investors (2018), “the foundations for a responsible exit are laid even before an investment is made. To increase the likelihood of continued impact after exit, investors often select investees based on whether impact is embedded in their business model or inextricably linked to financial success. They also seek to understand the likely growth trajectory of the business, which has implications for which exit paths and options will become available.” They also note that a “growth strategy’ is needed throughout and at (investment) exit is “a company’s continued access to the right resources, networks, and knowledge” for sustained impact.

The need for a thoughtful approach to sustainability is shown by Hiller, Guthrie, and Jones in “Overcoming Ex-Post Development Stagnation“ (2016). The authors cite “limited evidence of program efficacies coupled with government and agency preference for planning, approval, and implementation processes rather than sustainment of outputs, outcomes, and impacts means that ex-post performance, scaling, and sustainability is not well understood or well pursued…. [There is a] lack of willingness to commit time and resources to rigorous evaluation of post-project effectiveness“. This affects a vast number of projects. For instance, they found 63,000 projects in 2003 alone, and “relative to the number of development projects undertaken, ex-post project [evaluations] are not commonly carried out, meaning that rates of success are often unknown and the complexity of causalities and ex-post dynamics of interactions and processes are not well understood.“ This limits our learning from what has (not) worked and what to do more (or less) of, including those that could not be sustained with only local resources.

We make sustainability assumptions are participants long for them, as Valuing Voices also found. Hiller et al. state that “whilst project documentation commonly conveys an expectation that some process of spread will occur ex-post, it rarely does, despite strong ex-post case-study evidence of stakeholder requests for further development opportunities.” This cautionary feedback could mean some project activities could be so resource-intensive that they could not feasibly be sustained or spread without long-term support, and retaining results may be limited to less costly activities. Valuing Voices found that other activities could be remunerative enough (financially, in health or education outcomes, for instance), as to be locally demanded and continued to be pursued. We have found in our Valuing Voices research and the Tufts research that activities where incentives did not continue, tended to die ex-post, while those which continued to bring benefits, such as cash crops and credit, water supply, and health, were prioritized ex-post, even in the absence of external funding continuing.

Hiller et al. outline that there are multiple ways in which scaling-up environmental sustainability over time, over area and, interestingly, scaling-within projects. The Ex-post Development Stagnation authors are clear that “creating conditions to support longer-term sustainability beyond project completion represents a recurring challenge, and it is not uncommon for activities and institutions to become inactive ex-post or for stakeholders to revert to previously unsustainable practices.” They note that some watershed studies have even found that participants actively destroy project measures in some cases. Certainly, the inability of locals to sustain often expensive activities without a project or larger organization’s support is common. “If it is assumed that development needs remain outstanding, then there may be merit in ensuring that development projects do not just remain “isolated, one-time interventions, like unconnected dots on a white page” or “islands of salvation.” The authors concur, “based on project subsidiarity and participatory principles, scaling-within management should be devolved to the local level (local authorities and local communities) to allow communities and individuals to filter out irrelevant practices and encourage adaptation and evolution of activities which are of greatest perceived livelihood benefit”.

As Valuing Voices research on exit has shown, it is a process, not an event(forthcoming, with thanks to I. Davies). The Hiller article notes that sustainability is best enhanced by capacity-building during implementation and with time for handover where “organizations adopt modes of functioning that allow local communities and organizations to build conceptual, operational, and institutional capacities. While scaling-down does not mean that governments disengage from processes such as community-driven development — it does, however, require it to be more flexible and responsive to locally generated demand to ensure the terrain is fertile for community organizations to emerge, learn, and grow.“

Let’s work together to extend the sustainability of impacts. Would love to hear your thoughts…

“Impactful” Investments? Will investments change lives or just bottom lines?

 

“Impactful” Investments? Will investments change lives or just bottom lines?

 

Needs are growing in the developing world and blended finance, including Impact Investing, with the “intention to generate positive, measurable social and environmental impact alongside a financial return” could help amid pressures on philanthropy [1][2]. Investments look to fill the $4 trillion annual funding gap to fulfill the Sustainable Development Goals by 2030, as current donations to aid programs can only meet so much need. Such investment is rising. In 2018, investors interviewed by GIIN, an industry thinktank, managed over $228 billion in impact investing assets and more are aimed at SDGs. The Sustainable Development Goals (SDGs) are “a call for action by all countries – poor, rich and middle-income – to promote prosperity while protecting the planet. They recognize that ending poverty must go hand-in-hand with strategies that build economic growth and address a range of social needs including education, health, social protection, and job opportunities, while tackling climate change and environmental protection.” SDGS are to improve basic needs, natural capital, address climate change improve peace, equity and sustainability through a range of governance and empowerment interventions.

[3]

 

Impact Investing comes from a decade of ESG investments reorientation, based on these assumptions: “If environmental, social and governance (ESG) factors are ingrained in investors’ thinking, then a triple-win will follow – more valuable companies, more value for end-investors and, ultimately, a more sustainable system“ [4]. While there is a new focus on the 17 SDGoals by investing in enterprises in sectors such as agriculture, microenterprise, health and environment, investing broadly into sectors does not mean intended grassroots results are achieved (e.g. no poverty, no malnutrition). Far more tracking of the effects on the lives and livelihoods of those the SDGs aim to help is needed.

There is so much good business news, from companies investing in their own ‘circular economy’ in Asia to decrease waste, and carbon-neutral or forest-positive or health & justice initiatives and incubators in the USA, Europe and Africa [5][6][7][8]. There are positive clean energy examples such as large global investors in solar power which seem to have local effects [9]. A range of grassroots for-profit social businesses such as Greenlight Planet and Solar Sister, tangibly help address ‘energy poverty’ of two billion people with scarce access to basic and reliable electricity in developing markets [10]. Some crowdfunding is emerging through outlets such as Trine where micro-investors get a return for loan investing in poor communities’ solar projects which have proof of at least access to electricity [11]. Unfortunately, the depth of coverage is not sufficiently growing to reach the two billion given the climate change needs. Forbes reported that Toniic impact investor members have “$2.8 billion of impact investments, which are up from $1.65 billion in 2016 but the hype can be louder than proof of lives changed [2].

The funding is not yet living up to its promise as most investors are unwilling to sacrifice immediate profits for later returns, and what is called ‘patient capital’ with below-market returns may be needed to nurture those businesses in Asia, Africa and Latin America incurring some early losses while business models mature. GIIN in 2016, an industry tracker, notes that many investors lament risks being too high and businesses not having enough of a track record to justify investments [12]. Expertise from (I)NGOs working in global development can help. There is a growing risk of impact investment funds being more willing to invest in safe markets in North America and Europe, compared to higher risk emerging markets like Africa [13].

More perniciously, investments sold as being ‘impactful’ and addressing environmental sustainability, can include some wolves in sheep’s clothing (see italics): Axios 2019 revealed the following about “Green bonds” and an investment aimed at funding to help Ebola ravages:

  • “A recent World Bank bond was ostensibly designed “to raise awareness for the importance of investing in women and girls.” There’s no obvious way in which it does that, and it’s a minuscule $4 million in size — a drop in the bucket, compared to the Bank’s annual $50 billion in bond issuance….” [14]
  • “US municipal green bond issuance fell to $4.9 billion in 2018, per S&P Global Ratings, down more than 50% from $10 billion in 2017. Development banks also saw their green bond issuance fall. But for-profit banks, largely in China, took up a lot of the slack. They issued $48 billion of green bonds in 2018, up from $23 billion the previous year. The real impact: A lot of people who think that they’re making socially responsible investments are really just buying Chinese bank debt” [14]
  • “The World Bank’s Ebola bond was, it said, “a momentous step” that would “serve the world’s poorest people.” The idea: Were Ebola to flare up again in west Africa, investors’ money would be used to address it. Investors so far haven’t lost a penny, despite the outbreak of the 2nd-largest outbreak of Ebola on record.” [14]

What are the aims of impact investing if not tangibly having a social and environmental impact? Instead, a 2018 finance article, the drivers for investors are 1) risk management, 2) return potential, 3) mission alignment and 4) constituent/ stakeholder demand [15]. Much of the focus is on funding ‘checking the box’ of the 17 goals rather than showing results that funding changes people’s lives. While 76% of GIIN’s investors claim to “track their investment performance to the SDGs or plan to do so in the future,“ there is little data on how much their investments have decreased poverty, hunger, or increased literacy [16]. Sites like B-Analytics ‘track impact’ but actual changes to lives are assumed; Key Performance Indicators (KPIs) focus on business performance not on grassroots improvements [17].

I wanted to be delighted by Rockefeller Foundation’s Impact Investment Management (IIM)’s asset management platform that “aims to tap into mainstream markets and investors, scaling up investments into promising new finance vehicles that help to close the (SDG) funding gap” [17]. Yet they typify how impact investors misuse terms like ‘impact’. It is not synonymous with generic ‘results’, especially surprising given the strength of their M&E. IMM touts “Impact: The Rockefeller Foundation’s investment is expected to directly support 2,500+ loans to students“ [18]. As any monitoring & evaluation person knows, giving a loan is an input into their lives and is not an indicator of success. Recipients could drop out of school or fail to graduate, default on the loan, or conversely wildly succeed. We do not know ‘impact’ if we do not measure properly [19].


For those of us who care about results on the ground, among people in the ‘developing world’, so far ‘impact investing’ is still miles from outcomes and impacts that meet the aims of the SDGs, and more than ‘intent’ is needed in the 11 years until 2030. One think tank GALI screens results by headquarters location and “Impact Orientation: Indicates whether the organizations have the explicit intent of supporting ventures with social or environmental objectives” [20]. As a new Impact Investor Network Map notes, “a primary characteristic of “impact” is intentionality – the explicit pursuit of social impact“ and similar to the loan example above, ‘Access to Financial Services’ is the most frequently reported impact objective by investors“ [21]. Another source found that financial accelerator projects tracked are funded only 1-12 months, so those of us in global development wonder how much sustained impact could be generated in such a short time. Instead, some celebrate intermediate steps such as (good) new principles to foster environmental sustainability such as in the wild fisheries sector, and funding for U.S. forest restoration bond, which focuses on inputs/outputs only and principles are voluntary.

We at Impact Guild believe international and national non-profits and experts like us can fill this measurement expertise gap were funders interested. A recent report by Amplify, an international non-profit (INGO) consortium shows how these two communities can help each other to change lives [22].

While some impact investors are saying there aren’t enough viable investments with returns and track records, the INGOs are looking for that funding for the projects they know do work that affect lives on the ground. Challenges were: “identifying and acquiring funding’, ‘legal (and regulatory) barriers’, ‘communicating strategy’ and ‘organizational staff capacity’ to work in a for-profit manner to show ‘impact measurement’. As over 40 INGOs involved all indicated that delivering technical assistance was their favored approach, with ecosystem building and establishing social enterprises also considered by a majority, tracking results on the ground to fill this significant gap in current impact investor expertise is a logical bridge. As Mercy Corps Ventures said: “Early-stage businesses need more than just capital to grow and scale up their impact — they also need business and technical support to realize their vision. INGOs have unique assets that make them well-placed to provide such support. In fact, the real value INGOs add to investees comes through their post-investment engagement” as well as M&E expertise. So too do national research institutes, building real, national capacity for sustaining results.

Impact investors will need to create funding vehicles that have more ‘patient’ and smaller infusions of capital, and INGOs will need to focus assistance on enterprises and projects that are remunerative to investors. Further, INGOs noted the absorptive capacity of local ‘impact’ investments, as a median of only $37,000. Is this cost-effective for the for-profits to lend so little, even when ‘bundling’ investments into funds? Time will tell. While Development Impact Bonds supported by bilateral donors can fill a gap in confidence in more local, even riskier investments, the industry needs to focus on results on the ground before making claims of impact. Such funders, e.g. DFID and the new US government Development Finance Corporation (DFC) launching late 2019 can be the key [23]. Sorting out how much the investments will focus on poor countries, what will “development assessments” include, and how will USAID M&E expertise be used to look at actual, not just intended, “development impact” is vital.

Impact Guild remains hopeful that the for-profit sector, even aided by INGOs, can have actual grassroots impact. As Exponent Philanthropy said in 2016, “So much time and effort is focused on getting an impact investment closed and the funds disbursed. However, executing an agreement isn’t the finish line and, in fact, it may not even be the halfway point. Post-closing, there is still a lot of work to do in monitoring the performance of an investment to ensure financial health and intended impact” [24]. An array of metrics are being collected but often only to inform front-end due diligence on the investment bottom line rather than sustainable development results of lives changed. That gaping hole must be acknowledged and filled to start to meet the SDGs.

Do you agree?

 

 

Sources:

[1] OECD. (2017). The Private Sector: The Missing Piece of the SDG Puzzle. Retrieved from http://www.oecd.org/dac/financing-sustainable-development/development-finance-topics/Infographic%20-%20The%20Private%20Sector%20-%20Missing%20Piece%20of%20the%20SDG%20puzzle.pdf

[2] Dallman, J. (2018, December 31). Impact Investing, Just A Trend Or The Best Strategy To Help Save Our World? Retrieved from https://www.forbes.com/sites/jpdallmann/2018/12/31/impact-investing-just-a-trend-or-the-best-strategy-to-help-save-our-world/#3828adf775d1

[3] MSCI ESG Sustainable Impact Metrics. (n.d.). Retrieved 2019, from https://www.msci.com/esg-sustainable-impact-metrics

[4] Fraser, I. (2018, July 17). How ESG investing entered the mainstream. Retrieved from https://www.raconteur.net/finance/esg-investing-enter-mainstream

[5] Cheam, J. (2015, March 10). The rise of the circular economy in Asia. Retrieved from https://www.eco-business.com/news/rise-circular-economy-asia/

[6] Edie Newsroom. (2019, March 20). HP Pledges to Make All Printed Pages ‘Forest Positive’. Retrieved from https://www.edie.net/news/5/HP-pledges-to-make-all-printed-pages–forest-positive-/

[7] Edie Newsroom. (2019, March 19). Energy Catapult launches SME support for low carbon heating and cooling. Retrieved from https://www.edie.net/news/6/Energy-Catapult-launches-support-for-low-carbon-heating-and-cooling/

[8] Mpala, D. (2019, January 31). Here are 8 accelerators, incubators African startups can apply for in 2019. Retrieved from https://ventureburn.com/2019/01/accelerators-incubators-african-startups/

[9] CB Insights. (2017, December 6). The Solar Industry’s Most Active Investors. Retrieved from https://www.cbinsights.com/research/most-active-solar-startup-investors/

[10] Vella, H. (2018, July 19). Financing sustainable energy for developing economies. Retrieved from https://www.raconteur.net/finance/financing-sustainable-energy

[11] Lu, J. (2019, February 13). Crowdfunding the Sustainable Development Goals. Retrieved from https://www.undispatch.com/crowdfunding-the-sustainable-development-goals/

[12] Mudaliar, A., Pineiro, A., & Bass, R. (2016, December 7). Impact Investing Trends: Evidence of a Growing Industry. Retrieved from https://thegiin.org/research/publication/impact-investing-trends

[13] SGG Group. (2018, September 21). Impact investing in Africa. Retrieved from https://www.hvst.com/posts/impact-investing-in-africa-ERrTdDm3

[14] Salmon, F. (2019, March 17). Bill McGlashan’s firing exposes hypocrisy in impact investing. Retrieved from https://www.axios.com/tpg-capital-bill-mcglashan-impact-invsting-06828a4f-7390-46d8-b7e4-c6d3f7fbc7c7.html

[15] Digard, R. (2018, September 10). SDGs: Financial centres take cue from UN on sustainability. Retrieved from https://www.raconteur.net/finance/sdgs-financial-centres-sustainability

[16] Global Impact Investing Network. (2018, June 6). Annual Impact Investor Survey from the Global Impact Investing Network (GIIN) Shows Evidence Of Dynamic And Diverse Impact Investing Market (Press Release). Retrieved from https://thegiin.org/assets/2018%20Annual%20Impact%20Investor%20Survey_Press%20Release.pdf

[17] B Analytics. (n.d.). Demonstrate Your Leadership with a GIIRS Rating. Retrieved from https://b-analytics.net/giirs-funds

[18] Rockefeller Foundation. (n.d.). Impact Investment Management. Retrieved 2019, from https://web.archive.org/web/20190623144658/https://www.rockefellerfoundation.org/our-work/initiatives/impact-investment-management/

[19] Simister, N. (2015). Outputs, Outcomes and Impact. Retrieved from https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Outcomes-Outputs-and-Impact-7.pdf

[20] GALI. (n.d.). Accelerator Directory & Survey Methodology. Retrieved 2019, from https://www.galidata.org/accelerators/methodology/

[21] Impact Investing Network Map. (n.d.). Retrieved 2019, from https://impactinvestingmap.com/insights/

[22] Dente, T., Libby, S. B., Turpin, S. M., & Sharma, S. (Eds.). (2018, December 5). Amplify Impact Investing 2: The Next Mile of Impacting Investing for INGOs. Retrieved from https://www.humentum.org/impact-investing

[23] Saldinger, A. (2019, March 19). New US development finance institution edges toward launch. Retrieved from https://www.devex.com/news/new-us-development-finance-institution-edges-toward-launch-94497

[24] Velasquez, C. (2016, December 2). Impact Investing: Impact Measures and Monitoring Tools. Retrieved from https://www.exponentphilanthropy.org/blog/impact-investing-impact-measures-and-monitoring-tools/

 

Holiday Hallelujahs and my Wish List for Sustained Impact(s)

There are six bright Hallelujahs on the sustained impact journey this Holiday. I’ve paired each of them by a wish, so appropriate this Holiday Season.

1. Netherlands Greatness! The government of theNetherlands has recently started talking with their implementing agencies about how to track and prove sustainability six years after exit, as one Dutch colleague told me at the European Evaluation Society Conference this Oct.

WOW.

As we know, demanding evaluation of sustainability after project closure is rarer than vegan turkey on Thanksgiving, and the Dutch Ministry of ForeignAffairs is tacking the issues of a) how will they pay for it after project funding has ended (hint: see 3ie), b) how will the data disseminated and influence which decision-makers? Note: Japan’s development agency JICA is the only other bilateral to have this as a mandated aspect of their ongoing evaluations. We congratulate the Dutch for joining Japan in sustained impact evaluation leadership!

My Wish: For the other bilaterals and multilaterals for whom such evaluations are exotically occasional learnings to see the value from learning about the successes and failures of how(un)sustained your results are. Also to fund such evaluations for at least 10% of all programming per year, and do better programming focused on helping locals sustain the results they want. See, that wasn’t such a huge wish was it?

…

2. USAID Transformation to Self-Resilience is a promising analysis USAID is doing under the leadership of AdministratorMark Green. This Transformation “Journey to Self-Reliance” is “reorienting the way it does business to focus on supporting our partner countries… to solve their own development challenges.” It consists of a variety of policies, from using more private sector money for development to two exit-related initiatives including Financing Self-Reliance “to strengthen support for our host country partners in their efforts to finance their own development journey”and includes continuing to build local capacity and Partnering and Procurement such as using adaptive approaches and “principles of supporting locally-led development and long-term sustainability.”

Will USAID really follow the SDG aim to Leave No one Behind in terms of focusing on the poor? Metrics of USAID’s Country Roadmaps, range from national economic growth and democracy/governance to sectoral measures in health, food security etc. Maybe this is not just a way to use national data to justify exit of aid programs due to shrinking funding, but instead is a way for USAID to determine who is actually self-reliant among their recipient countries and focus more resources on those who aren’t on more needs-based criteria than politically strategic bases than in the past.Maybe it will look hard at assumptions rather than Uganda’s CDCS which projects sustainability in 30 years but will program on until then. OXFAM’sAid Localization may be a better, faster route.

My Wish: that bilateral donors like USAID test how well their current aid projects support self-reliance and how well they’re furthering sustainability. This includes learning what has been self-sustained and not renewing what hasn’t. We offer our database of 25 catalytic organizations’ ex-post project evaluations at Valuing Voices which were informed by local voices of our participants and partners. These fascinating evaluations need to inform new programming as sustainability is only high occasionally, unlike what we assume. We ask commissioners of such evaluations to tell us what they have learned and how they have done things differently (e.g. have they funding differently, chosen what projects are selected as implementers differently, and while adaptive monitoring is a terrific addition, how has evaluation changed -other than pending OECD DAC Criteria and Ofir’s blog commentary plus the WorldBank’s sustainability blog) and how has exit, sustainability planning and handover to national government using data about sustainability and impact changed and people really can sustain themselves? NOTE: this may lead to less money being spent over a longer time, more flexibly and locally-directed. We donor nations may benefit less as locals take their own ‘development’ in hand. Hallelujah.

…

3. Recently I did a webinar for InterAction webinar with dear sustained impact aficionados from Catholic Relief Services and World Vision on “How Sustained and Resilient are our Impacts?” Lessons about accountability for locally sustainable development was front and center. We all shared that ex-posts project evaluations have, shall we say, mixed results of success and failure. This can be difficult for agencies to absorb as the projects selected are not only typically selected to showcase success, but normally these are selected to be showcased as the best projects. Also, unsustained results can question the quality of funder’s design the agency’s and implementation, risking reputation and funding for both. But hiding failure is worse. A recent Evidence Action statement admitting failure of a promising intervention, promising to evaluate why and learn from it only gained kudos. Conversely, great results can show where future focus could be, or much can be (rightly?) attributed to unexpected new conditions which led it in very different directions.

My Wish: My CRS colleague was challenged by a younger American colleague who asked him “are we liable if we don’t deliver sustainable development?” Astonished, he answered “yes.”. Let’s get over our infallibility and inclination to promise something only the countries themselves can deliver. Yes, we are liable for wasting money and locals time and hopes. Yes, we can claim success when partnerships go well. May we all learn from failures, celebrate successes –especially those locals consider successes, and replicate only those.

…

4. Impact Investment is slowing inching forward in helping move more of the Sustainable Development Goals (SDGs) toward fruition.  The pressure on many global funders and countries alike to meet the 17 SDG targets set in 2015 is rising.  Funding needs to accomplish the goals is $4 trillion a year, yes, trillion, compared to $150 billion available now in foreign aid. Accessing private capital is a wise investment if such development could unleash up to $12 trillion in growth. Repeatedly people like myself and Impact Guild read that “demand for SDG-aligned investment products outstrips supply“ which is true not in terms of the actual business and civil society non-profits trying to generate Social, Environmental, Governmental and financial Return on Investment (SROI and ESG), but in terms of profitable and stable investments.  Investors tend to be risk-averse yet there are a range instruments that cover the range of development to humanitarian needs. It is not a mechanism to ignore as for-profit funding will sharply increase in coming years with up to $8 trillion of impact investments moving into global disasters recovery alone and vital “climate finance” investments only going to grow, as $100 billion are year is being raised for poor countries to stem the rising effects of climate change. Combined with a $10.3 billion private Green Climate Fund that privately funds climate resilience projects all over the world to meet SDG13, we can no longer rely on diminishing foreign aid alone.

While patient pessimism is needed to overcome the hype that now private capital can replace philanthropy, INGOs are – cautiously innovating. As this month’s Amplify report from 40 international NGOs dipping their toes into impact investment collaborations notes, there are a wide array of collaborations. Some are investing their own private donors funds – or those leveraging other donors’ and investors’ funds in Development Impact Bonds (DIBs) or Social Impact Bonds (SIBs), and other instruments to the tune of $545.1million in assets. These are mainly in profit-low-hanging fruit development sectors of agriculture, livelihoods, and financial inclusion. Yet some of the INGO’s greatest strengths are least appreciated and what the relatively risk-averse and ‘due-diligence-focused’ impact investors do not yet know they need: ”their deep knowledge of local environments, programs, and technical solutions; their long-standing networks and sophistication in partnering with multiple actors; their financial sophistication from their wide-ranging donor relationships; and their experience in complex, multi-year measurement of impact.“

My Wish: that these two industries listen to one another. Non-profits can be mired in a can’t spend-can’t-risk mindset while being queasy about generating revenue, while impact investors can be torn between profits being king and claims of ‘impact’ from just investing in a sector without monitoring actual effects on the ground. There is simply too much good to be done.

…

5. Brazil/ France’s Corporate Social Responsibility (CSR) – The French renewable energy company Voltalia has created their first CSR-funded Social Projects for Social Responsibility. It is using some excellent sustainability principles as well as leveraging Brazilian Bank BNDES social sub-credit seed financing. I met Voltalia’s small CSR team at the EES conference. They started working in impoverished northeast Brazil, using great non-profit processes of communities identifying needed projects and processes that overlapped with what CSR could offer, over half of which were in education, health and ‘social responsibility’ (aka livelihoods linked to water access and use such as fisheries etc). They invested substantial funds into these grassroots projects, alongside interesting Social Management Tools that included cost management tracking of local co-investment, great collaborative communications plans, sustainability reporting, including Social Return on Investment (SROI). Like many non-profits before them, it has been hard, but also enormously rewarding and their work has led to greater employee engagement and was recognized by global headquarters – kudos! – interested? Contact them.

My Wish: Walking in the shoes of our partners and participants teaches us about their capacities and needs, and how to adapt what we can offer to be of greatest use. We need more companies with ‘dust on their shoes’, as my PhD advisor called my fieldwork. This epitomizes the Sidekick Manifesto approach where we offer local leaders help, rather than ‘solving’ poverty, hunger, ill-health, un-representation for them (which our efforts only accomplish for a short-term anyway, see #1, 2).

…

6. Climate Conference COP24 corralled nearly 200 nations to create technical targets and measurements to try to keep us below IPCC’s dire projections that we are not on target to limit the temperature increase to 1.5C degrees, and that it will cost billions to the economies of the world, not to mention jeopardize past development gains. You may wonder why this is a Hallelujah. It is because for those I most care about – my children and future generations-especially those in the ‘less developed countries’ who have least-caused yet are least able to mitigate climate change effects- are finally starting to be seen and heard. Conferences such as COP24, as well as a huge range of ‘non-state actors’, including corporations, non-profits, philanthropies, tribes, even individuals like you and me.  Whether it is our self-interest or expenditure-avoidance which propels us to decrease our emissions and increase our use of clean technologies, or our altruism toward the millions of species on earth, we have globally begun turning to sustainability of the ecological kind. Scientists are proposing remarkable inventiveness to capture already present CO2, studies are showing how we can be more energy efficient, buy more sustainable fashion, to eat more kindly, etc.

My Wish: We have begun changing our consumption and carbon emission and our youth are pushing us to conserve our planet. I am hopeful that we will come together to address this threat to all species. Is it far enough? Is it fast enough? Santa – help! So since it’s my holiday hallelujah, read Hans Rosling on how it’s all getting better anyway.

Happy Holidays and New Year everyone!