by Jindra Cekan | May 16, 2018 | Accountability, Aid effectiveness, CGDev, Evaluation, ex-post evaluation, Feed the Future, Feedback Labs, foreign aid, impact investing, Impact Investors, International aid, international development, Local Participants, Participants, Participation, post project evaluation, post-project evaluation, Return on Investment (ROI), SALT, Sidekick Manifesto, SUSRoi, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Sustained and Emerging Impacts Evaluations (SEIE), USAID, Valuing Voices
Setting a higher bar: Sustained Impacts are about All of us
Global development aid has a problem which may already affect impact investing as well.
It is that we think it’s really all about us (individuals, wealthy donors and INGO implementers) not all of us (you, me, and project participants, their partners and governments). It’s also about us for a short time.
All too often, the measurable results we in global development aid and Corporate Social Responsibility (CSR) funded projects that last 1-5 years track and report data for two reasons:
1) Donors have Compliance for grantees to meet (money spent, not lost, and results met by fixed deadlines of 1-5 years – look at some of the European Commission Contracting rules) and
2) Fund recipients and the participants they serve are accountable to ‘our’ donors and implementers who take what happened through their philanthropic grants as ‘their’ results.
Both can skew how sustainably we get to create impacts. An example of such strictures on sustainability from USAID. As respected CGDev Elliot and Dunning researchers found in 2016 when assessing the ‘US Feed the Future Initiative: A New Approach to Food Security?‘ the $10.15 billion leveraged $20 billion from other funders for disbursement over three years (2013-16). “We are concerned that pressure to demonstrate results in the short term may undermine efforts to ensure any impact is sustainable…. Unfortunately, the pressure to show immediate results can encourage pursuit of agricultural investments unlikely to be sustained. For example, a common response to low productivity is to subsidize or facilitate access to improved inputs… it can deliver a quick payoff… however, if the subsidies become too expensive and are eliminated or reduced, fertilizer use and yields often fall…..
With so much focus on reporting early and often about the progress in implementing the initiative, there is a risk that it increases the pressure to disburse quickly and in ways that may not produce sustainable results. For example, for 2014, Feed the Future reports that nearly 7 million farmers applied “improved technologies or management practices as a result of U.S. Government assistance,” but only 1,300 received “long-term agricultural sector productivity.” Are the millions of others that are using improved inputs or management practices because of subsidies likely to have these practices sustained? And how likely are they to continue using improved practices once the project ends?”
3) Impact investors stick to the same two paths-to-results and add a new objective: market-competitive financial returns. They also need to show short-term results to their investors, albeit with social, environmental and governance results like non-profits (future blog).
4) Altruists create things we want ‘beneficiaries’ (our participants) to have. For instance a plethora of apps for refugees cropped up in recent years, over 5,000 it is estimated, which can be appropriate, nor not so helpful. Much like #2 above, ‘we’re’ helping ‘them’ but again, it seems to be a ‘give a man a fish’… and my fish is cool sort of solution… but do our participants want/ need this?
How often is our work-for-change mostly about us/by us/ for us... when ideally it is mostly about ‘them’ (OK, given human self-interest, shouldn’t changes we want at least be about all of us?).
All too often we want to be the solution but really, our ‘grassroots’ clients who are our true customers need to generate their own solution. Best if we listen and we design for long-term sustainability together?
As the Brilliant Sidekick Manifesto stated in two of its ten steps:
a) “I will step out of the spotlight: Sustainable solutions to poverty come from within are bottom-up, and flow from local leaders who are taking the risks of holding their politicians accountable and challenging the status quo.”
b) “I will read “To Hell with Good Intentions” again and again: Politicians, celebrities and billionaire philanthropists will tell me that I can be a hero. I cannot. The poor are not powerless or waiting to be saved. Illich will check my delusions of grandeur.”
We have examples of where we have stepped away and participants had to fend for themselves. At Valuing Voices, we’ve done post project-exit evaluations 2-15 years afterward. What did participants value so much that they sustained it themselves (all about them, literally)? These Sustained and Emerging Impacts Evaluations (SEIE) also give us indications of Sustained ROI (Sustained Return on Investment (SusROI) is a key missing metric. As respected evaluator Ricardo Wilson-Grau said in an email, “I think calculating cost-effectiveness of an intervention’s outcomes would be a wonderful challenge for a financial officer searching for new challenges — if not a Nobel prize in economics!”)
Most of these evaluations are pretty bad news mixed with some good news about what folks could sustain after we left, couldn’t and why not. (These are the ones folks expect to have great results, otherwise they wouldn’t share them!) While most clients are understandably interested in what of ‘theirs’ was still standing, and it was interesting disentangling where the results were attributable by implementation or design or partnership flaws or something else, what was mesmerizing was what came from ‘them’.
The key is looking beyond ‘unexpected’ results to look at emerging impacts that are about ‘them’ (aka what we didn’t expect that was a direct result of our project, e.g. spare parts were no longer available to fix the water well pump once we left or a drought rehabilitation water project that decreased violence against women), to what emerging results are attributable not to use but only to our participants and partners who took over after our projects closed. One example is a Nepalese project ended yet the credit groups of empowered women spawned groups of support groups for battered women. Another is a child maternal health project changed how it worked as women reverted to birthing at home after NGOs left; community leaders punished both parents with incarceration in the health clinic for a week if they didn’t given birth there (wow did that work to sustain behavior change of both parents!).
Many of us at Valuing Voices are shocked that funders don’t seem that interested in this, as this is where they not only take over (viz picture, sustaining the project themselves), but they are making it theirs, not ours. Imagine assuming the point of development is to BE SUSTAINABLE.
Source: Community Life Competence
Our participants and national stakeholder partners are our true clients, yet… Feedback Labs tell us Americans alone gave $358 billion to charities (equivalent to the 2014 GDP of 20 countries) – in 2014 but how much of this was determined by what ‘beneficiaries’ want? Josh Woodard, a development expert, suggests a vouchers approach where our true clients, our participants, who would “purchase services from those competing organizations… [such an] approach to development would enable us all to see what services people actually value and want. And when we asked ourselves what our clients want, we would really mean the individuals in the communities we are in the business of working with and serving. Otherwise we’d be out of business pretty quickly.”
This opens the door to client feedback – imagine if participants could use social media to rate the sustained impacts on them of the projects they benefited from? A customer support expert wrote in Forbes, “Today, every customer has, or feels she has, a vote in how companies do business and treat customers. This is part of a new set of expectations among customers today that will only grow ... you can’t control product ratings, product discussions or much else in the way of reviews, except by providing the best customer experience possible and by being proactive in responding to negative trends that come to the surface in your reviews and ratings stronger.”
So how well are we working with our participants for ‘development’ to be about them?
What do you think?
by Jindra Cekan | Mar 6, 2018 | Accountability, cambodia, civil society, disabled, international development, NGOs, Sustainability, Sustainable development, Youth
The Name of the Game is “Sustainability” but Does the Last Player Count?
by John Lowrie, Reblog from https://www.scribd.com/document/11621086/The-Name-of-the-Game-is-Sustainability-but-Does-the-Last-Player-Count-by-John-Lowrie
Today, it is obligatory to answer the question “how will your proposed activities be sustainable after the project?” Most of us dutifully play the game and repeat various sentences that describe the measures that “should” bring sustainability about. Then it is usually left at that. We all move on to the next project and donor. Seldom does anyone look back over the passing of many years to check to see if promise has turned in to reality. At best there may be an end-project evaluation which will say that planned outcomes are achieved and “likely” to be sustained. It would be a very brave evaluator to be more committed beyond that.
My question is why do we play this game? (I should add: apart from the obvious answer we need the donors’ money!) Sustainability is much more than a ruse in a game. It goes to the very essence of what kind of organisation we belong to; plus all the others with us whether staff, supporters or beneficiaries; how we originated and what are our real long- term plans?
Many NGOs are artificially created groupings of people, unlike the first ones last century that emerged as people on a mission to address specific issues that they felt strongly about. Latterly, NGOs are now lumped in to “Civil Society” which is another recent fashionable label in development jargon. This is despite many officials and even NGO workers in Cambodia having no idea that it is supposed to encompass more than NGOs, to embrace other groups such as the press, trade unions, etc.
Many NGOs have actually been started by or for charismatic individuals, tapping in to somebody else’s cause or source of funding, rather than part of a collective if not mass movement in its own right towards a common end. Sometimes, international departing NGOs promote them, as part of “exit-strategies” to demonstrate that they leave something behind to show for their good work and all the money spent. They may have even included a plan for it in their sustainability proposal write-up. In Cambodia the “gravy train” that accompanied
UNTAC, the massive international effort to bring lasting peace and democracy in the early 90s, spawned many. Some of these NGOs have been good and stood the test of time. Many have fallen by the wayside. Others have been co-opted by political interests. Few have proper accountable self-governance structures.
◄“It’s the money that matters”, but will these NGO beneficiaries, the final players, get what they need to be sustainable?
So how can they be really sustainable? What is driving them – apart that is from the obvious – money? It is true that many do good work. They often do work that local authorities should be doing. But, I ask, towards what end? For example “poverty alleviation” is not an end in itself; for it to be sustained it needs much more than the usual 1-3 year time-frames that donors favour for their projects. Yet the usual pattern is –identify your target groups, go there, pass on whatever to them, then move on (to the next ones). There are exceptions. Lutheran World Federation (LWF) expect to work for 10 or more years in their target communities before they “graduate” and begin a systematic staged withdrawal. Mostly, however, the projects end on time or soon after, the files are closed, and that is that, at least until next time.
Meanwhile NGOs try to source repeat funding or new funding, and the game rules dictate where and how they proceed. It is only by luck rather than design that there will be a true match between what they want to do and what the donor is willing to give the money for. Usually there is accommodation, most likely on the part of the NGO; indeed some big donors explicitly rule out comments, queries, and changes to their guidelines which become grails of holiness. The most desperate NGOs have to re-invent themselves; depart from their original mission, suddenly acquire new skills in often far removed fields to stay in the game. I have seen an election-monitoring organisation become an agency to consult displaced families affected by a road-widening. I know of one local rural development NGO in one province become a pro-citizen governance campaigner in another. Adaptability and learning to diversify are good qualities, but when they cause such radical changes within an NGO, it cannot be re-assuring for sustainability.
The nature of what an NGO does and its underlying philosophy is therefore key to sustainability. Those NGOs created to work in the immediate post-conflict or disaster emergency relief periods are prone to short-term visions, and sometimes they leave legacies which handicap development such as “dependency” and even “easy-come-easy- go” attitudes when foreign money seems plentiful. Those NGOs born in the next phase, i.e., after emergency relief, the start of infrastructure reconstruction and restoring public services and the economy, can also be equally short-term in their vision. In fact they have an inherent flaw. If they succeed, they do themselves out of a job and few want that as it would mean no future jobs and income! Even the Cambodian Government, despite an addiction to foreign aid, now maintains that some have overstayed their welcome.
These NGOs tend to be “welfare” or “service-provision” providers. They are confident in their abilities “we know best” but are they committed to passing their best skills and knowledge on and to the right people to take progress forward without them?
Cambodia is not alone in that politics plays a big part. The ruling party now has a 73% majority in the National Assembly, holds 98% of the 1,629 Commune Council Chiefdoms and 70% of Councillors. As the national and commune members form the constituency for the Senate, Provincial, and District Authorities, the party is guaranteed monopoly control. The situation is not helped by the absence of a neutral civil service or public service – in fact it is mystifying that UNTAC and every international donor has not tried to cultivate one. So we are left with what we have; we are where we are. It means that sustainability will only follow if we accept that reality. We have to engage constructively with the powers-that-be; we have to find the best people we can to work with, and through them, reach out to others. If not, eventual opposition or just plan lack of good will and support will affect the final outcome. Authorities always outlast NGOs, at least in Cambodia they do!
Cambodia did not have a good start when NGOs first came on the scene. Many of the first NGOs were human rights activists needed at that time and to a certain extent now, to expose calamitous treatment meted out to victims. Unfortunately this profile, still high in perceptions, has made many in power believe that this is only or mainly what NGOs do and stand for1. The legalistic approach to human rights where abuses are reported, perpetrators identified, and “name, blame shame” attached, cannot be a development tool in the sustainability tool-kit. It must be separated out and equally important alternative human rights approaches such as “rights-based development” which arouse less hostility co-exist with similar enthusiasm and means. NGOs and civil society will never have sustainable activities while their undoubted overall positive contribution gets little or no recognition by the people that count most when it comes to change.
NGOs are agents of change, which if mishandled does lead to suspicion, so to achieve change, NGOs must be clear in their message to persuade all (or a majority) that their change is worthwhile. If a long-term, wide cross-section commitment towards that change is not implanted, it is not sustainable. Too many development initiatives in Cambodia have resulted in temporary change. The “status-quo” reverts soon afterwards. In some cases the change is only tacitly accepted “take their money”, “go along with them” but “once they have gone, we go back to how we were, OK?” This is the opposite of sustainability.
There have to be contextually appropriate solutions, which can only be country-by- country, culture-by-culture. We should not have to operate on the basis of big international donors with their “one-size-fits-all” development policies and calls for proposals that allow just one format (theirs) that automatically favours the big international NGOs. They have their professional fund-raisers, who can knock out proposals that score high marks in the assessment/evaluation boxes, without many of the authors and assessors concerned ever going near the intended target beneficiaries! Yet be under no doubt, the words they pen and peruse answer beautifully on how such beneficiaries were involved, but who ever checks?
I may not be a good team-player in the present development game, so what am I suggesting as an alternative? First of all I would like to see fewer NGOs, ones which are smaller, more self-contained, and manageable to operate country-by country, sector-by- sector, region-by-region. They need to have good links to others elsewhere for best practice to be shared, but their core mission needs to be focussed to bring about certain defined changes; with the right people and the resources they need, and in the fullness of time. Therefore 5 years is an absolute minimum and asking for “core costs” to be provided should not be regarded as a mortal sin, as it tends to be with most donors. It has to be allowed, to be seen as value for money, and though indirect, still an essential element in bringing about that change. If not, how can sustainability be served?
Ironically, it is only with core support or independent means, that NGOs can play the game, so again favouring the big players. How else does an NGO cover its costs in preparing bids to donors? For example after UK-DfiD released their worldwide Global Transparency Fund in 2008, 450 organisations applied, and just 38 succeeded. Even a recent in-country release by UNDP for environmental projects attracted no less than 67 applications, of which just 13 won out. This means that for 412 in the DfiD case and 54 in the UNDP case who had devoted considerable effort, on top of their normal work, the process ultimately proved to be a waste of time; to be disappointing and especially for local NGOs to be discouraging.
I have two (NGO) organisations in mind while writing this article. Neither conforms to the pattern I criticize. They are different and I am not the only one to think that they are better. Both have poor disabled people as their target beneficiaries. One is in sports. The other is in poverty alleviation/new livelihoods or careers and self-advocacy. Both are now purely local NGOs, with none of the trappings or spending power of the big international players that dominate the disability sector. In fact neither has anything like a long-term future, because of funding gaps, and so a lot of their work, despite excellent results so far, cannot be said to be sustainable. If they go out of business too soon, how will their beneficiaries stay involved, if sustainability is to be realised eventually?
Local NGO “Cambodia National Volleyball League Disabled2” is the sports one. Most interestingly, it is taking a route party by choice, partly by necessity, towards the private sector or “corporate social responsibility” funds in its hope to realise sustainability. CNVLD has earned a worldwide reputation for transforming the self-esteem of disabled athletes. They actually enjoy high standing on the world stage unlike their “able-bodied” compatriots. They play volley-ball, pursue wheel-chair racing and some athletes may well qualify for the 2012 London Paralympics, even if there may be no money to support them to go and compete there.
◄Disabled Sports Athletes of NGO CNVLD playing the game at its best – but do they have a future?
Now CNVLD’s quest is to raise such money and cover their modest (compared to INGO) costs. Yet this laudable aim is treated with derision by some in the sector. Pursuing private funding is viewed as an anathema to many who see it as contradictory to the “not- for-profit” concept. Does this make sense? Does such obstruction make for sustainability? Is it a sin not to want to depend solely on the usual institutional sources of funding? One critic is interesting. It has in its mission that all its services to beneficiaries must be provided free. No charges whatsoever must be levied even for those who can afford to pay! In Cambodia, everybody pays, even when services are supposed to be free3. Even if a direct charge is not levied, recipients expect and are expected to show their gratitude4, and do so especially the poorest who see it as an inescapable obligation. Surely sustainability means (a) people who can afford to pay should do so, and (b) NGOs who can attract funding and not depend on taxpayers’ money should be welcomed5?
The second organisation, New Horizons Society6, is not as lucky as CNVLD, as they cannot go down the route of sponsorship which is an accepted feature of sport. This is an NGO that did localise from an INGO but on their own terms. They voted against joining another national body created by and favoured by the big international disability NGOs. Over 200 of their Focus Persons (Group Leaders) voted in a secret ballot to form their own NGO in order to stay true to their close-knit grassroots upwards origins and growth. Now they have 3,175 members in 135 self-help groups federated up to provincial level and going on to the national stage. They have remarkable accomplishments in creating new livelihoods for their [once] ultra-poor members and have accumulated over $130,000 in revolving funds. Individual lives have been transformed. One boy has gone from beggar in the market to national singing celebrity. One young man went from lonely at home; never seen a computer; no English, to being one of today’s high-flying geeks. His class-mate, from a similar start, went on to become the Publicity Officer for CNLVD and to lead her own troupe of dancers in the NHS Child Advocacy Group performing at international conferences.
◄NGO Child Advocates demonstrate “We can do” but will donors let them?
The 135 groups went from fear of talking to officials to successful advocacy. They started with the right to education and health-care for disabled youngsters, and once they were confident went on to persuade ministers to take action to stop their meagre pension rights being denied to them. These people chose not to ask for pity, or welfare, or for service provisions to be given to them. Instead they ask simply for the chance to show that “they can do” and that they can be self-sufficient when given the opportunities and means.
Their problem is that they cannot do this for all members yet, let alone go on to include others in the same fate as they once were. Yet despite their accomplishments, right now they can only win project activity funds. Donors refuse to pay more than 20% for running costs. Some specify as little as 5%. That does not even cover the running costs of their multi-purpose centre where meetings, training, sports, dancing, computer classes etc., go on. It may stretch to pay modest salaries to 2 or 3 staff, but they have to depend on consultant/advisors like me to help them voluntarily. Their entire organisation is radically different from the familiar set-up to be seen in international disability, donor and development organisations. There is not an air-conditioner, land-cruiser, or voucher paying university fees of expatriate’s children, etc., in sight. Yet despite its low cost and high yield, it is not yet sustainable within the present rules of most donors. If it can get over its current shortfalls and continue to build the revolving funds in to a sizeable trust or investment fund, with many members whose incomes have grown sufficiently able to subscribe fees to run their organisation, then they may have an independent viable future. But is there a donor who can adopt such a long-term vision; who will give enough to meet the real needs they identify, and who will then stay the course even when inevitable setbacks happen on the way?
So finally sustainability should be expressed in one simple notion and by the last player. It should be a measure of the change made in individual lives and over life-times of beneficiaries and their families. They are the last players in the game – who else but them can make that assessment? The present game means that it is the other players that have most say. They arrive on the scene much earlier; play in their own compact time-frames, and to their own rules. Then as external entrants, they depart the scene as soon as they can. Sustainability?
_____________________________________________
Footnotes:
1 The perception is not helped by the limitations of the Khmer language. When the word “advocacy” was first introduced “tasumateh” was used, literally “to struggle for” associated with confrontation, not partnership, and so it was viewed by many as an alien Western concept.
2 info@standupcambodia.net or www.standupcambodia.net
3 Understanding pro-poor political change:the policy process Cambodia by Caroline Hughes and Tim Conway, Overseas Development Institute, 2003 – DfiD Publication.
4 It is this “tradition” that is at the heart of a current dispute within the UN-backed Khmer Rouge
Trials, where Cambodian staff are alleged to have paid a proportion of their salary to officials with a role in their appointment.
5 Subject of course to disclosure and transparency.
6 nhs@camintel.com or www.newhorizonsunlimited.org
by Jindra Cekan | Jan 8, 2018 | foreign aid, GIIN, impact investing, Impact Investors, international development, post project evaluation, Return on Investment (ROI), stakeholders, Sustainable development, Valuing Voices
Impact Investing – International Development’s New Holy Grail?
There are so many things I love about the private sector such as Forbes 18 Dec Quote of the Day: “You’re going to be wrong a fair amount of times. So the issue is, how do you be wrong well?” asked Ray Dalio, Founder of Bridgewater Associate. This is a key issue for impact investors and international ‘developers’ alike.
International development suffers from the myth that failure must be downplayed. Too often only success is highlighted, whereas project shortcomings are framed as: “less successful” “numerous issues affected a less optimal…” Yet by downplaying the less great (Aka awful) results we miss vital learning that private sector expects, learns from and integrates toward the greater success. Why? Many in foreign aid believe (rightly?) such admissions might endanger winning more funding for more projects. Even as recently as 2014, U.S. foreign aid industry websites such as DevEx are still posting: “One can be forgiven for forming the impression that our development efforts are nearly perfect if typical annual reports, scientific conferences and event social media content are the basis for information. Successes are proudly packaged in glossy formats and heavily disseminated, whereas any objectives not achieved are relegated to the obligatory, and typically short, lessons learned section. This practice does not accurately represent an important reality: development efforts do in fact fail” [1].
Admitting failure, posting failure reports are awfully rare in international development, but how bad is it? The Asian Development Bank wrote in a large overview of the sustainability of post-project results, “Some early evidence suggests that as many as 40% of all new activities are not sustained beyond the first few years after disbursement of external funding” [2]. A 2017 Cambridge University study found that “using an original database of over 14,000 small development projects in Ghana, I estimate that one-third of projects that start are never completed, consuming nearly one-fifth of all local government investment” [3]. Even when they do start, complete, and even have salutary results at the end of the project, Valuing Voices research shows quick declines toward failures in as little as two years post exit, such as these post-project results at the AEA 2017 conference. The foreign aid industry is so focused on showing results while conditions are (relatively) conducive, that far fewer than 1% of all projects are evaluated for what was still standing in as little as two years after project closeout, and those are mostly those projects expected to be successful. Sustainable, long-term results suffer from what CGDev researchers are concerned “that pressure to demonstrate results in the short term may undermine efforts to ensure any impact is sustainable….Unfortunately, the pressure to show immediate results can encourage pursuit of agricultural investments unlikely to be sustained” [4]. Luckily there’s a place to go. DevEx reminds us that “Venture capitalists and corporate investors understand that less than 20 percent of new businesses will succeed,” hence my love of the private sector’s admitting, learning and improving that ‘aid’ needs [1].
As a former investment banker (Solomon Brothers) and management consultant (Price Waterhouse & Coopers and Lybrand), I know that the corporates care for results, and do not shy away from pulling money from where things don’t work and put it where they do. 30 years in international development showed me that rigid bureaucracies and fixed ‘project cycles’ and an industry focused on ‘getting money out the door’ lead to a focus on accounting for all funds, but not for changing lives over the long term. Virtually no one calculates return on (our) investment compared to the cost of projects, especially including the value of what projects generate and participants can sustain.
I am quite fervently hoping Impact Investors focused on financial ROI to firms and investors as well as Social Return on Investment will step in, fund gathering and learning from the whole range of ‘returns’. Will they share both financial profits/ losses and feedback from the whole social ‘value chain’ of stakeholders of those involved on what succeeds and fails? Will investors learn from national partners and participants on what should be done better? If yes, all of us will win. I am heartened by cautiously optimistic statements such as Next Billion’s “a core characteristic and challenge of impact investing is the measurement and management of social and environmental impacts alongside financial returns. Development cooperation and impact investing communities can build on their respective experience in results measurement and learn from and with each other” [5]. We can IF we are going to the same place.
From my early look at impact investing, it is a ‘game changer’ with $250 billion in assets looking for a profitable home [6]. UBS Asset Manager Baldinger says “In the past you sold products to your client, now you empower your client to create a desired impact. As an industry, we’ve had to rethink everything we do — impact and sustainability is the Silicon Valley of finance and we want to be the Google” [6]. These are happy words to someone focused on sustained (and emerging) impacts but among impact investors, so far, ‘impact’ seems to be thrown about as specifically as ‘results’, and GIIN ‘sustainability’ metrics are so wide ranging as to illuminate less quality than quantity. So far, much of their metrics look more like outputs relevant to companies (‘clients served’, ‘new investment capital’) that results of SROI. While there is something to be said about measures of ‘organizations trained’, ‘poverty assessments’ done, at least as a start, yet does ‘gross profit’ indicate that corner of the world is better off (and does this measure the investment into the enterprise, or is this of the investment fund itself)? Does ‘communities served’ and ‘social impact objectives’ illuminate the quality of the impact on lives changed? Is anyone asking how long-lasting, and sustained these investments, measuring what I call SUStained Return on Investment (SUSROI), will be after these investors leave (which is what I suspect most investment participants and millennial investors think they’re buying)?
This is the start of a series of blogs exploring how we who care about generating and evaluating sustained impacts can learn from, inform, (gasp) shape impact investing’s gargantuan footprint in international development. Powerhouses such as the Rockefeller Foundation, Ford Foundation and Soros are looking, teaching, investing, and all public and private equity as well as a whole range of other investors now invest in this new hybrid [7][8]. Who else is? What can we learn to make the world better? What do you think: Is impact investing development’s holy grail?
Sources:
[1] Petruney, T. (2014, December 12). Facing global development’s fear of failure. Retrieved from https://www.devex.com/news/facing-global-development-s-fear-of-failure-85078
[2] Asian Development Bank. (2010, October 31). Post-Completion Sustainability of Asian Development Bank-Assisted Projects. Retrieved from https://www.adb.org/documents/post-completion-sustainability-asian-development-bank-assisted-projects
[3] Williams, M. J. (2017). The Political Economy of Unfinished Development Projects: Corruption, Clientelism, or Collective Choice? American Political Science Review, 111(4). Retrieved from https://www.cambridge.org/core/journals/american-political-science-review/article/political-economy-of-unfinished-development-projects-corruption-clientelism-or-collective-choice/1351C9A6EB64B39B0D3A2B0A2D748412
[4] Elliott, K. A., & Dunning, C. (2016, March 1). Assessing the US Feed the Future Initiative: A New Approach to Food Security? Retrieved from https://www.cgdev.org/publication/assessing-us-feed-future-initiative-new-approach-food-security
[5] Next Billion. (2017, November). Financing Global Development – Leveraging Impact Investing for the SDGs. Retrieved from https://nextbillion.net/calendar/financing-global-development-leveraging-impact-investing-sdgs/
[6] Kennedy, E. (2017, December 18). Impact investing: A $250 billion game-changer for finance. Retrieved from https://www.cnn.com/2018/09/27/investing/impact-investing-wall-street-banks-asset-managers/index.html
[7] Ford Foundation. (2017, April 5). Ford Foundation commits $1 billion from endowment to mission-related investments. Retrieved from https://www.fordfoundation.org/the-latest/news/ford-foundation-commits-1-billion-from-endowment-to-mission-related-investments/
[8] Karabell, S. (2013, August 14). Impact Investing, Soros-Style. Retrieved from https://knowledge.insead.edu/responsibility/impact-investing-soros-style-2576
by Jindra Cekan | Mar 2, 2017 | Aid effectiveness, Better Evaluation, Corporate Social Responsibility, Dare to Lead, Effective Philanthropy, ex-post evaluation, Foundations, impact evaluation, Impact Investors, international development, Participants, partners, post-project evaluation, SEIE, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Valuing Voices
Leading in Challenging Times:
Sustained and Emerging Impacts Evaluation (SEIEs)
Some American organizations are retrenching, focusing more attention on domestic rather than international programming. Some are pulling back from critique of international development to informing legislators of its benefits; the Center for Global Development’s changed ‘Rethinking US Development Policy’ blog to only “US Development Policy“. UN’s Refugee Agency questions whether to challenge Washington’s tough line on refugees from countries such as Syria, or should it stay quiet in the hopes of protecting its funding [1]?”
Reticence is understandable in this ‘climate’, so to speak, but fear does not change the world, leadership does. Envisioning and creating the world we want gets us there.
There may be no better time to build the evidence base on what works in sustainable development as these are low cost investments if we use national staff and focus research well. We have seen this in the fewer than 1% of all projects that have been evaluated post-closeout for sustainability [2]. At the very least, we can learn what we should do differently in the next design, to fully foster sustainability, once more funding emerges. Many are interested in great results. Hundreds of ‘impact evaluations’ are happening on aid effectiveness; our industry wants to learn what works and what we could do better.
Our SEIE work goes beyond current understanding of ‘impact’ to see what projects our partners and participants can self-sustain ex-post for years to come which is an excellent investment in proving cost-effectiveness. While some governments’ investments can diminish in the short term, national governments, and other funders such as a range of international bilateral and multilateral donors, foundations corporate social responsibility and impact investors do want to invest in provably “sustainable” development [3].
Why should we invest in SEIEs?
- Hundreds of thousands of projects are still being implemented.
- Millions of participants are still hoping what we are doing together will be sustainable.
- Billions of dollars, euros, kwacha, pesos, rupees are being spent on new projects that need to be designed and implemented for future sustainability.
Implementing organizations could be fearful to see what remains once funding and technical assistance are withdrawn, but such a view not only robs our industry of exciting lessons on what did change and was so valued that it was sustained, but also what to not do again. Not returning post-project also short-changes our participants. In our SEIEs, we have found participants and partners creating new ways to carry on, innovating beyond what we could imagine during our assistance.
We also need to start now to design and implement for sustainability. doing SEIEs, we can start to understand the ‘drivers’ behind the Sustainable Development Goals (SDG) results with countries tracking some 120 indicators across 17 goals. Currently countries are tracking up to 230 indicators across the 17 Goals [4]. But while such monitoring shows ‘GDP has increased or ‘under-nourishment has decreased’, there is little or no information on what has caused it. Yet doing and SEIE on a large donor-funded programme, we can explore what elements made projects sustainable and how to do more (or less) there and elsewhere. Such sentinel site support for learning about sustained and emerging impacts is key to understand some of the why, for example, did income or health improve.
Dare to lead, especially in these challenging times. We know of organizations that are doing these evaluations internally, others are publishing them on their sites. Leadership happens at all levels, from internal, technical to managerial and administrative work to external evaluators and consultants as well as public pressure.
How can you foster sustained impact?
- You can advocate for such evaluations
- You can share the SEIE guidance, below, and start to design and implement, monitor and evaluate sustainably in all projects/ proposals you are designing now.
- You can see if your organization has done any post-project sustainability evaluations and we can post them on Valuing Voices’ repository, celebrating your organization.
We can help you learn how to do these. Our partner, Better Evaluation, just published our Sustained and Emerging Impacts Evaluation as a ‘new’ evaluation ‘theme.’
Guidance there shows you [5]:
1. What is SEIE?
2. Why do SEIE?
3. When to do SEIE?
4. Who should be engaged in the evaluation process?
5. What definitions and methods can be used to do an SEIE?
Resources
References
SEIEs will grow as will examples, discussions, and joy as embracing sustainability sprouts, and sends us progressing in yet-unforeseen ways! We are excited to be in the final stages of receiving a research grant to further guide SEIEs. We will share that news in our next blog.
We want to learn from you:
- What do you think needs to be in place for funders to move beyond the funding cycle and do an SEIE?
- What would help to make this type of evaluation more widely undertaken?
- If you have done a post-project evaluation, how did you do it? What were some of the barriers you faced and resources you were able to draw on to overcome them?
How can we lead together to Value the Voices of those we serve!?
(Reposted from https://medium.com/@WhatWeValue/leading-in-challenging-times-sustained-and-emerging-impacts-evaluation-seies-617b33bf4d27#.ec7fcg4ty)
Sources:
[1] Foulkes, I. (2017, February 27). Is there a US diplomacy vacuum at the UN in Geneva? Retrieved from https://www.bbc.com/news/world-europe-39080204
[2] Cekan, J. (2015, March 13). When Funders Move On. Retrieved from https://ssir.org/articles/entry/when_funders_move_on
[3] UN DESA. (2011, March 2). Lasting impact of sustainable development. Retrieved from https://www.un.org/en/development/desa/news/sustainable/sustainable-development.html
[4] UN Statistics Division. SDG Indicators: Global indicator framework for the Sustainable Development Goals and targets of the 2030 Agenda for Sustainable Development. Retrieved March, 2017, from https://unstats.un.org/sdgs/indicators/indicators-list/
[5] Cekan, J., Zivetz, L., & P, R. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE
by Jindra Cekan | Dec 12, 2016 | Aid effectiveness, DFID, Doing Development Differently, Evaluation, Exit strategies, Feedback loops, Food for Peace (FFP), International aid, international development, NGO, Sustainability, Sustainable development, Sustained and Emerging Impact Evaluation (SEIE), Sustained Impact Evaluation, USAID, Valuing Voices
Embedding Sustainability Everywhere – All Five Slices Now
It has been a tumultuous year, and next year does not look like we will have much stability as a respite. As domestic concerns grow larger in two huge economies, US and UK, the question of the place foreign aid will play abound in conversations around the world.
However 2017’s transitions transform our work, for now I do have some good news,
1) Sustainability can be cheap. Far cheaper, in fact to design for sustainability, create feedback loops checking on sustainability through the eyes of our partners and participants, monitor for sustainability than to assume it’ll happen and far cheaper than finding out funds could have had far greater impact if we had valued their voices in the first place.
In our work to help our clients and partners fund, design, implement and monitor/ evaluate with sustainability in mind, we created what we hope is a helpful tool (guidance forthcoming).
a) By Designing for Sustainability with those who will sustain them, their financial buy-in and commitment are far higher (see CRS/ Niger [1]), as is advocacy and community buy-in (see new post-project OXFAM/ DRC [2]) and there are indicators the costs of start-up later are more cost-effective.
b) By clarifying Sustainability Indicators we check assumptions about who will do so, how much of a priority our activities are before we scale them up (Federation/ Ethiopian Red Cross). Retrospective post-project sustainability evaluations also enable us to learning from past successes and do better.
c) Sustainability Monitoring and Adaptation involve those pesky but pivotal feedback loops which are vital to understanding if we have gone off the rails or not, especially in terms of unexpected shocks derailing logical frameworks of designed projects. USAID’s nice recent CLA (Collaborating, Learning, Adapting) process includes donor funding for adaptation mid-stream which fosters effectiveness and sustainability [3]. Even lovelier are the Doing Development Differently examples that are often very low-cost and high-effect.
d) Informed Exit, Stakeholder Sustainability Consultation should be done throughout the cycle, at least a year earlier than most projects can begin this (note; not the last few months please). Transitioning for success leverages sunk costs for ongoing results. It includes heavy knowledge management on how the implementers managed information and resources, tracked data, sustained outputs and outcomes which now local partners will need to do, etc (FFP Exit Strategies study, a Czech study, and a new UK Results study shows range of items to consider during and post-transition) [4] [5].
e) Post-Project Sustainability: Our Valuing Voices/ Better Evaluation/ Tufts joint presentation at AEA on SEIE has more on how learning from post-project evaluation lessons can change sustainability of future projects for the better [6]! Further, what capacities and systems have been built in-country to sustain the results after our funding and expertise leaves? What can we do differently?
Can it be done? Demand is rising. I recently presented at a conference about embedding sustainability in programming now, and an enterprising NGO took this idea to heart. They proposed joint design with communities, which is the very bedrock of designing for sustained impact. Kudos!
2) Secondly, donors are willing to pay for sustainability. Sustainability is ensconced in USAID/ Food For Peace’s (FFP) 2016 Strategy and CLA (above) [7]. In the section called New learning and Implications for FFP Programming, they present findings such as “actions that drive big results during the life of the project may actually undermine sustainability in the long run. It raises the question as to whether FFP is willing to accept more modest results in the near term if they can be delivered in a way that will yield more sustainable gains over time….”
They also point us in the direction of country-led development: “Sustained capacity, resources, motivation, and linkages all require a focus on catalysts for change beyond FFP. Facilitative approaches that rely on and strengthen local actors help ensure that resource and knowledge transfers, and the incentives and linkages that support them, will be self-perpetuating beyond project end”. Notably, while UK’s DFID focuses on maximizing impact through Value-for-Money, it is a shorter-term economy, efficiency and effectiveness rather than sustained impact for the end-users and DFID’s exit strategies have recently been critiqued [5].
There is an issue of disincentives for the new administration to heed (if the agricultural lobby for US food exports does not prevail): “In [USAID’s exit] study evaluating how sustainable the results of Title II development programs are 2–3 years after project closure, FFP found that “providing free resources can threaten sustainability, unless replacement of those resources both as project inputs and as incentives has been addressed” [8]. As the Natural Resources section notes, “Whether entirely in the hands of the community or linked to a formal institution, the incentives and resources necessary to maintain a community asset are part of the system that will sustain it. The lack of such systems is visible in rusted irrigation pumps, failed mangrove plantations, abandoned bore wells, eroded dikes, and silted-in fish ponds around the world” [7]. Yet the private and public sectors are important: “Sustainable, broad-based change is more likely to be achieved by supporting and strengthening existing community, private sector, and public sector mechanisms for product and service delivery, and by supporting the capacity, quality, and accountability of government institutions” [7].
FFP’s new strategy calls for taking a systems approach to change that emphasizes sustainable long-term gains over unsustainable short-term wins. Even more delightful, in a small meeting at MFAN with Dina Esposito, Director of Food For Peace in November, she announced that they were looking to do pilot funding of an additional three years to typical five-year DFAP development projects. One year would involve collaborative participatory design between partners, communities and FFP, the second additional years being evaluation post-project of sustained and emerging impact!
This is a sea shift that can hopefully withstand political winds. After all, US foreign aid accounts for less than 1% of our federal budget, even though many Americans believe it is over 15% (hence easy to cut)…. but fingers crossed the aid effectiveness value of our work is… Valued.
Sources:
[1] Cekan, J., PhD, Kagendo, R., & Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all
[2] Lindley-Jones, H. (2016, November 16). ‘If we don’t do it, who will?’ A study into the sustainability of Community Protection Structures supported by Oxfam in the Democratic Republic of Congo (DRC). Retrieved from https://policy-practice.oxfam.org.uk/publications/if-we-dont-do-it-who-will-a-study-into-the-sustainability-of-community-protecti-620149
[3] USAID Learning Lab. (n.d.). Collaborating, Learning, and Adapting (CLA)? Retrieved from https://usaidlearninglab.org/faq/collaborating%2C-learning%2C-and-adapting-cla
[4] Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
[5] Del Mese, F. (2016, November 16). When aid relationships change: DFID’s approach to managing exit and transition in its development partnerships. Retrieved from https://icai.independent.gov.uk/report/transition/
[6] Cekan, J., Rogers, B. L., Rogers, P., & Zivetz, L. (2016, October 26). Barking Up a Better Tree: Lessons about SEIE (Sustained and Emerging Impact Evaluation). Retrieved from https://valuingvoices.com/wp-content/uploads/2016/11/Barking-up-a-Better-Tree-AEA-Oct-26-FINAL.pdf
[7] USAID. (2016, October 6). 2016–2025 Food Assistance and Food Security Strategy. Retrieved from https://www.usaid.gov/ffpstrategy#:~:text=FFP’s%20new%20strategy%2C%20the%202016,USG)%20food%20assistance%20as%20a
[8] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
by valuingvoicesjin | Nov 1, 2016 | American Evaluation Association (AEA), Evaluation, Exit strategies, Food for Peace (FFP), Food security, impact evaluation, international development, Project cycle, SEIE, Sustainability, Sustainable development, Sustained and Emerging Impact Evaluation (SEIE)
Presenting Lessons on (post-project) Sustained and Emerging Impact Evaluations from the U.S. AEA Conference
Dear readers, attached please find the Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation presentation we did last week at the American Evaluation Association (AEA) conference in Atlanta GA [1]. I had the pleasure of co-presenting with Beatrice Lorge Rogers PhD, Professor, Friedman Nutrition School, Tufts University (aka the famous Food for Peace/ Tufts Exit Strategy study [2]), Patricia Rogers PhD, Director, BetterEvaluation, Professor, Australia and New Zealand School of Government (where we recently published guidance on SEIE [3]) and Laurie Zivetz PhD, International Development Consultant and Valuing Voices evaluator.
We integrated our presentations from Africa, Asia and Latin America into this fascinating overview:
1.Sustained and Emerging Impact Evaluation: global context
2.SEIE: definitions and methods
3.Case studies: findings from post-project evaluations
4.Designing an SEIE: Considerations
5.Q&A — which fostered super comments, but since you couldn’t come, please tell us what you think and what questions you have…
There are amazing lessons to learn about design, implementation, M&E from doing post-project evaluation. We have also grown in appreciating that sustainability can be tracked throughout the project cycle, not just during post-project SEIE evaluation.
We’ll be building this into a white paper or a … (toolkit? webinar series? training? something else?). What’s your vote ___? (I know in this US election season, so… :)).
What would you like to get to support your learning about Sustained and Emerging Impact Evaluations? Look forward to hearing from you- Jindra@ValuingVoices.com
Enjoy!
The full presentation is available here:
https://valuingvoices.com/wp-content/uploads/2016/11/Barking-up-a-Better-Tree-AEA-Oct-26-FINAL.pdf
Sources:
[1] Cekan, J., Rogers, B. L., Rogers, P., & Zivetz, L. (2016, October 26). Barking Up a Better Tree: Lessons about SEIE (Sustained and Emerging Impact Evaluation). Retrieved from https://valuingvoices.com/wp-content/uploads/2016/11/Barking-up-a-Better-Tree-AEA-Oct-26-FINAL.pdf
[2] Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
[3] Zivetz, L., Cekan, J., & Robbins, K. (2017, May). Building the Evidence Base for Post-Project Evaluation: Case Study Review and Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
