Assuming Sustainability and Impact is Dangerous to Development
(+ OECD/ DAC evaluation criteria)
We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impact such as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.
INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions” [1]. They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact” [1]. Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer. Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable” [2].
How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.
Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).*
There are remarkable assumptions routinely made. Many projects put sustainability into the proposal, yet most close out projects in the last 6 months. Rarely do projects take the time to properly phase down or phase over (unlike CRS Niger); many exit ceremonially ‘handing over’ projects to country-nationals, disposing project assets, and leaving only a final report behind. Alternatively, this USAID Uganda CDCS Country Transition Plan which looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit [3]. Maybe they will measure progress towards that goal and orient programs toward handover, as in the new USAID “Journey to Self-Reliance” – we hope! Truly, we can plan to exit, but only when data bears out our sustained impact, not when the money or political will runs out.
As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters” [4]. In fact, most ‘impact evaluations’ actually test effectiveness rather than long-term impact. Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.
if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment [5][6]. These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!
We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants. We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership” [7]. Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms” [7]. It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…
[3] USAID. (2016, December 6). USAID Uganda Country Development Cooperation Strategy 2016-2021. Retrieved October, 2018, from https://www.usaid.gov/uganda/cdcs
[6] Ofir, Z. (2018, October 13). Updating the DAC Criteria, Part 11 (FINAL). From Evaluation Criteria to Design Principles. Retrieved from https://zendaofir.com/dac-criteria-part-11/
Setting a higher bar: Sustained Impacts are about All of us
Global development aid has a problem which may already affect impact investing as well.
It is that we think it’s really all about us (individuals, wealthy donors and INGO implementers) not all of us (you, me, and project participants, their partners and governments). It’s also about us for a short time.
All too often, the measurable results we in global development aid and Corporate Social Responsibility (CSR) funded projects that last 1-5 years track and report data for two reasons:
1) Donors have Compliance for grantees to meet (money spent, not lost, and results met by fixed deadlines of 1-5 years – look at some of the European Commission Contracting rules) and
2) Fund recipients and the participants they serve are accountable to ‘our’ donors and implementers who take what happened through their philanthropic grants as ‘their’ results.
Both can skew how sustainably we get to create impacts. An example of such strictures on sustainability from USAID. As respected CGDev Elliot and Dunning researchers found in 2016 when assessing the ‘US Feed the Future Initiative: A New Approach to Food Security?‘ the $10.15 billion leveraged $20 billion from other funders for disbursement over three years (2013-16). “We are concerned that pressure to demonstrate results in the short termmay undermine efforts to ensure any impact is sustainable…. Unfortunately, the pressure to show immediate results can encourage pursuit of agricultural investments unlikely to be sustained. For example, a common response to low productivity is to subsidize or facilitate access to improved inputs… it can deliver a quick payoff… however, if the subsidies become too expensive and are eliminated or reduced, fertilizer use and yields often fall…..
With so much focus on reporting early and often about the progress in implementing the initiative, there is a risk that it increases the pressure to disburse quickly and in ways that may not produce sustainable results. For example, for 2014, Feed the Future reports that nearly 7 million farmers applied “improved technologies or management practices as a result of U.S. Government assistance,” but only 1,300 received “long-term agricultural sector productivity.” Are the millions of others that are using improved inputs or management practices because of subsidies likely to have these practices sustained? And how likely are they to continue using improved practices once the project ends?”
3) Impact investors stick to the same two paths-to-results and add a new objective: market-competitive financial returns. They also need to show short-term results to their investors, albeit with social, environmental and governance results like non-profits (future blog).
4) Altruists create things we want ‘beneficiaries’ (our participants) to have. For instance a plethora of apps for refugees cropped up in recent years, over 5,000 it is estimated, which can be appropriate, nor not so helpful. Much like #2 above, ‘we’re’ helping ‘them’ but again, it seems to be a ‘give a man a fish’… and my fish is cool sort of solution… but do our participants want/ need this?
How often is our work-for-change mostly about us/by us/ for us... when ideally it is mostly about ‘them’ (OK, given human self-interest, shouldn’t changes we want at least be about all of us?).
All too often we want to be the solution but really, our ‘grassroots’ clients who are our true customers need to generate their own solution. Best if we listen and we design for long-term sustainability together?
As the Brilliant Sidekick Manifesto stated in two of its ten steps: a) “I will step out of the spotlight: Sustainable solutions to poverty come from within are bottom-up, and flow from local leaders who are taking the risks of holding their politicians accountable and challenging the status quo.”
b) “I will read “To Hell with Good Intentions” again and again: Politicians, celebrities and billionaire philanthropists will tell me that I can be a hero. I cannot. The poor are not powerless or waiting to be saved. Illich will check my delusions of grandeur.”
We have examples of where we have stepped away and participants had to fend for themselves. At Valuing Voices, we’ve done post project-exit evaluations 2-15 years afterward. What did participants value so much that they sustained it themselves (all about them, literally)? These Sustained and Emerging Impacts Evaluations (SEIE)also give us indications of Sustained ROI (Sustained Return on Investment (SusROI) is a key missing metric. As respected evaluator Ricardo Wilson-Grau said in an email, “I think calculating cost-effectiveness of an intervention’s outcomes would be a wonderful challenge for a financial officer searching for new challenges — if not a Nobel prize in economics!”)
Most of these evaluations are pretty bad news mixed with some good news about what folks could sustain after we left, couldn’t and why not. (These are the ones folks expect to have great results, otherwise they wouldn’t share them!) While most clients are understandably interested in what of ‘theirs’ was still standing, and it was interesting disentangling where the results were attributable by implementation or design or partnership flaws or something else, what was mesmerizing was what came from ‘them’.
The key is looking beyond ‘unexpected’ results to look at emerging impacts that are about ‘them’ (aka what we didn’t expect that was a direct result of our project, e.g. spare parts were no longer available to fix the water well pump once we left or a drought rehabilitation water project that decreased violence against women), to what emerging results are attributable not to use but only to our participants and partners who took over after our projects closed.One example is a Nepalese project ended yet the credit groups of empowered women spawned groups of support groups for battered women. Another is a child maternal health project changed how it worked as women reverted to birthing at home after NGOs left; community leaders punished both parents with incarceration in the health clinic for a week if they didn’t given birth there (wow did that work to sustain behavior change of both parents!).
Many of us at Valuing Voices are shocked that funders don’t seem that interested in this, as this is where they not only take over (viz picture, sustaining the project themselves), but they are making it theirs, not ours. Imagine assuming the point of development is to BE SUSTAINABLE.
Source: Community Life Competence
Our participants and national stakeholder partners are our true clients, yet… Feedback Labs tell us Americans alone gave $358 billion to charities(equivalent to the 2014 GDP of 20 countries) – in 2014 but how much of this was determined by what ‘beneficiaries’ want? Josh Woodard, a development expert, suggests a vouchers approach where our true clients, our participants, who would “purchase services from those competing organizations… [such an] approach to development would enable us all to see what services people actually value and want. And when we asked ourselves what our clients want, we would really mean the individuals in the communities we are in the business of working with and serving. Otherwise we’d be out of business pretty quickly.”
This opens the door to client feedback – imagine if participants could use social media to rate the sustained impacts on them of the projects they benefited from? A customer support expert wrote in Forbes, “Today, every customer has, or feels she has, a vote in how companies do business and treat customers. This is part of a new set of expectations among customers today that will only grow ... you can’t control product ratings, product discussions or much else in the way of reviews, except by providing the best customer experience possible and by being proactive in responding to negative trends that come to the surface in your reviews and ratings stronger.”
So how well are we working with our participants for ‘development’ to be about them?
Can’t wait to learn from post-project sustainability evaluation? If not why?
A colleague who has been promoting ex-post sustainability evaluation in her organization questioned my claim that doing them had “benefits” for future programming. It was an “untested assumption that there will be sufficient, strong enough evidence to apply to future programming… [and] the need to have evidence to cite for future work is not pressing enough.”
If you are on aid’s receiving end, what you care about is that good results are sustained, and you are able to live better, longer. You might want to show others evidence of what was sustained, rather than only what worked while external investments were there but stopped since. Absolutely, aid donors need to have evidence that something designed, funded, implemented, and monitored & evaluated showed good results, but we assume our results will be sustained after we have closed out and moved on. How well have we done? Let’s see.
At the American Evaluation Association meetings this month, several post-project evaluations were presented. Some came from Valuing Voices research, some from Social Impact, PLAN and World Vision and some others [1].
Results 3-5 years post close-out were, shall we say unexpected, from CRS Madagascar:
and Burkina Faso, where MCC/ PLAN found that three years post project “BRIGHT still had a significant positive impact—6.0 percentage points for children between ages 6 and 22—on self-reported enrollment. The impacts are smaller than estimated impacts on enrollment at 7 and 3 years after the start of the program,” they were rare [6].
If we don’t wonder why things didn’t work or why they did, and don’t return to find out if it happened again and what to do/ not to do again?, Often we continue to do very similar programming elsewhere, again assuming great results. How can we close our eyes and not do post project, Sustained and Emerging Impacts Evaluations (SEIE) and see, learn, do better? How can we continue to do very similar water/ sanitation, health, food security, and education programs and projects (with potentially similar results), and call ourselves sustainable development professionals? Shouldn’t we always ask not how effective is our aid when it’s there, but after its gone?
Building the Evidence Base for Post Project Evaluation:
A report to the Faster Forward Fund
We are delighted to share Valuing Voices’ report on the value added of post-project evaluation, which compares findings from eight end-of-project and subsequent post project evaluations [1]. Many of you are aware of how rarely post project evaluations are undertaken. As a result, there is little real evidence about project impact on long-term sustainability. Valuing Voices received a grant from Michael Scriven’s Faster Forward Fund to begin to address this gap.
Our findings show that post project evaluations can contribute to better understanding of sustainability impacts, and reveal unexpected and emerging outcomes years after project close. They also indicate ways in which we can design and implement for sustainability.
Finding suitable projects for this review was difficult because so few post project evaluations are done, fewer are publically available, and fewer still had comparable final evaluations and included local voices. Agencies that fund post project evaluations offer a range of reasons for doing so: to learn, to promote a success, to inform replication or scale, to provide justification for future funding, to promote accountabilities. However, many funding agencies consider post project evaluation a luxury or not necessary. JICA and OECD are notable exceptions in this regard.
Highlights include:
The review highlights the range of methods that have been used in post project evaluations, and point to the advantages of planning for sustainability measurement from the outset of the project.
The cases reviewed in the study highlight the (sometime dramatic) difference between the anticipated trajectory of a project, what is happening as the project ends, and what actually continued, was adapted, ceased or changed course after close out.
Taxonomies, knowledge management about evaluation, data retrieval/ retention, analysis, use and dissemination are elements of sustained impact evaluation that require attention.
Little documentation is available about how post project evaluations have actually informed and influenced organizational learning, sectoral dialogue or future programming.
Post project evaluations shed particularly interesting light on what emerged post-project that was entirely due to the efforts and resources of participants and partners after project investments stopped. More on these Sustained and Emerging Impacts Evaluations (SEIEs) at Better Evaluation.
We welcome your comments on this report and checklists, and encourage you to share it in your networks and get us feedback on their use. Please use the report and findings to advocate for more post project sustainability impact evaluations which will contribute to greater evidence-based learning about project sustainability. Valuing Voices is among a handful of organizations who do post-project evaluations and we can either conduct one or refer you to another who does.
Thank you,
Laurie Zivetz, MPH, PhD and Jindra Cekan, PhD, with Kate Robins, MPH, PhD of Valuing Voices
Poor villagers like Edith, Aminata, Rituu, and Juan don’t appear much on the nightly news. You might never know they exist unless you stop and read your mail from some charity asking you to help them. On the brochures, they can look scared or sad; maybe surrounded by their thin children, with a parched land or dying animals behind them. Our foreign aid programs should be helping them, but are they?
I have met these people while they are working in their fields, growing corn and peas, millet and coffee, raising their chickens and goats. I have talked to them outside of health centers where they have brought their babies to be vaccinated or their parents for medical care. I got to know them when I interviewed them under the big tree in the middle of their village, or in empty school rooms, asking them what they need from us, and how we can design foreign aid projects to better help them.
I have worked in international development as a technical expert in project design and monitoring and evaluation for international non-profits such as Catholic Relief Services, Save the Children, the Red Cross, and many others including the Bill and Melinda Gates Foundation and the US Agency for International Development (USAID). I estimate that I have designed and evaluated over 200 projects in 28 countries over the last 29 years. I have felt lucky to do this work, and foreign aid does achieve some very good work while we are there: helping farmers to farm better, or helping men and women to care for their family’s health, wealth, and future with new knowledge, tools, and items they need for daily living. My colleagues do wonderful work as well, in hard conditions, within countries with few resources, and for donors with unrealistic expectations of how much can be done well in short timeframes. In 2010, USAID stated that they would aim for 30 percent of funding to be spent by national partners under USAID Forward.1 This is an excellent step toward the country-led development that the Paris Accords promised, yet as of 2016, there is no list of local partners, other than a handful of examples.2 The only ‘country partners’ list posted to the USAID website includes 80 organizations doing programming via USAID in Afghanistan alone, 55 of which are American firms, four US agencies, nine Afghan government-affiliated organizations, six foreign governments, six UN agencies, and two MENA firms.3 Not quite the national civil-society-NGO partners we envisioned in 2010. Under the new U.S. administration, these are likely to shrink even more as the 0.5 percent of our GNP we allocate to foreign aid is redirected inward—that much more reason to make it as sustained as possible. European aid as well as other rising world nations need this approach just as much.
Large parts of international aid system remain broken. We design too many projects outside of the countries themselves. We have fixed funding and leave in pre-set times rather than when participants are actually ready to take over. We ‘handover’ without partnering throughout the whole project so that partners can determine what they are able to sustain. Even worse, we leave and do not look back to learn from our Ediths and Juans after our projects have closed. Sometimes, we disparage their knowledge, and at other times we don’t make enough time to ask but wish we did. Mostly, our aid industry is designed around measuring success while we intervene, and then abruptly leave because funding ended. Yet development is, as international evaluator Ian Davies says, “A process, not a result.”
Our policies say we are doing “sustainable development”, that we are helping our ‘beneficiaries’ (really our partners and participants) feed themselves over the long-term, that our projects are almost all successful, and that all we need to do is to scale up the great projects out there. But the numbers prove we are not, in fact, achieving sustainable development. Nine times out of ten, we rarely go back to talk to our participants and partners after our project end, and we move on.
The numbers are staggering:
Of the US$5 trillion dollars of international foreign aid spent since 1945, we have evaluated the long-term sustainability far less than one percent of the time.
Since 2000, for example, USAID and the Millennium Challenge Corporation have only done three such evaluations apiece, yet they spent well over US$300 billion.
The EU evaluated only a few dozen of its projects and programs, in spite of spending US$1.5 trillion in the last 15 years. The United Nations Development Program may do up to six a year, and the World Bank more, but how often do any of them talk to project participants and design anew based on what we learned that succeeded and failed?
The Japan International Cooperation Agency, and to some degree EU bilateral countries (through the Organization for Economic Cooperation and Development), have evaluated the sustainability of over 300 projects.
Despite this, tens of thousands of new projects are launched every year.
Satellites atop homes in a slum in Tigray, Ethopia.
This is why I founded Valuing Voices—to analyze what little we know to make development better.4 Not to destroy international development, but to change how we fund, design, and implement it. We need to design for sustainability of the activities by the country nationals themselves, rather than designing for results we can show to get more funding. We also must jointly implement, monitor, and evaluate our projects so countries can continue after we leave. Smaller organizations can do even simple activities, designing projects based on what the participants feel they can self-sustain, and partnering with those who will take over while they are still there.
Having spread the word for the last three years, to mostly little response, I now turn to you, readers. Our analysis – and a wonderful 2012 book, Time to Listen by Mary Anderson and Dayna Brown, shows that sometimes when our projects partner with country nationals, their people become – and stay – better off.5They want to be engaged, yet our very structure of delivering aid prevents this. Often we are not there long enough to make a lasting difference, or we invest scarce time on untested innovations that work in some places but don’t in others. Even worse, sometimes we design activities so badly that villages are left with irrelevant technology and trainings, wasted funding, and lost hopes. At other times, there are successes as well, but not returning robs us of the chance to replicate those. We do ‘impact evaluations,’ but only on successes during project implementation, and not on what people can self-sustain after we leave. Our vision is so limited. Our well-meaning self-interest blinds us.
Across the board, our development projects make one massive and incorrect assumption that once we ‘handover’ the project, the local government, community, and households have the means to sustain our multi-million dollar investments.6 We assume that technical knowledge will still be locally available to the villagers, that inputs like seeds and tools, data and vaccines will be accessible both physically and financially, that the government staff have the means to get to villages or that new NGOs and donors will appear to fill the gaps. The Huffington Post has stated, “as long-term projects and action-plans are established, more investment must go into financing locally designed solutions and projects that ensures ownership is placed back into local communities.”7 While more project have begun using feedback loops of listening to participants during implementation, virtually all good work stops when project funding stops.
Don’t we want development to be sustained after resources leave, and the opinions of these aid recipients to be heard? Don’t we want the next project to address the needs better? Don’t we, as taxpayers, want to demand that agencies using our tax dollars learn what is really sustainable and what is not? And shouldn’t we demand that all projects costing more than US$1 million over the past 10 years be examined now for lessons learned by sector (agriculture, health, credit, education, etc) and region? Shouldn’t post-project sustainability evaluations be included in all new projects? Don’t our participants and partners deserve the dignified futures they hope for, our creating channels for their voices that enable them to evaluate us and teach us how we can help them to be successful?
In fact, a radical Foreign Aid Transparency Act was just passed in the U.S. in June 2016.8 The bill calls for the President, within 18 months of enactment, to “set forth guidelines…for the establishment of measurable goals, performance metrics, and monitoring and evaluation plans that can be applied with reasonable consistency to covered United States foreign assistance.” These include ‘ex-post’ (sustainability) evaluations, and “can have enormous value when it comes to making programming and budgeting decisions.” Yet while there is a call for guidance to be developed, no funding came along with this bill. Without the funds to make this happen, this may be more ‘window dressing’ for sustainable development than excellent policy.
A small enterprise in Cap Haitien, Haiti.
There is some hope coming from the corporate sector. While impact investors are often more focused on return from emerging economies than fostering sustainable development, corporate social responsibility is building bridges in lovely ways. Tsikululu Social Investment of South Africa has thought about what advice to give to the companies they advise on such investments, as well as exiting from them.9
We argue that our budgeting needs a basic business metric: Return on Investment. In a time of huge demands on our resources worldwide from refugee flows, terrorism and climate change, we currently do little or no analysis of:
How much actual investment: What percent of allocated funds went to the activities that benefitted the partners and participants themselves, rather than being used as overhead for operations?
How much return: What is the value of what remains used 3-10 years after we leave? What was the value-added that communities and other funders (including the national governments themselves) who followed catalyzed based on our earlier investments? What were unexpected new results that emerged?
We envision a beautiful future, one where Edith, Aminata, Rituu, and Juan and their local partners are at the center of development. We imagine a world in which we listen to what people in need can sustain for themselves. Through these approaches data is shared widely on what has worked best and why; aid projects invest in country-systems and staff that boost their ability to self-sustain; and only sustainable projects are designed and funded that foster country-led development. The global adoption of the Sustainable Development Goals finally puts the focus on all we can do to foster sustainability of our work.
By valuing voices and focusing on sustainable solutions for excellent impacts, this will promote truly sustainable development from our aid organizations, government and non-governmental alike. We have much to learn, and there is not a moment to waste before we start Valuing Voices of those we serve and partner with on country-led development.
During this research, a senior international development evaluation expert told us that they can’t return to evaluate now-closed projects because they aren’t the same projects anymore (after closeout) and we are no longer responsible for the results. That took my breath away.
All new projects come from old projects… we recycle old project design most of the time, occasionally making substantive changes in targeting or design but much of how we design and implement remains the same. And while we thoroughly evaluate them during implementation, learning ex-post is a key missing link which all projects in the future can benefit from as we do similar interventions and track similar outcomes year after year but we rarely know which ones were sustained or emerged anew. There absolutely are aspects that get adapted but there are only so many ways to heal the sick, improve crop growth, save money, learn to read and so on, and there is a world we need to learn about what enabled some to be sustained and even morph into new results!
This excellent article, Do NGOs (non-governmental organization) help?, notes that “due to donor pressure [NGOs] are increasingly forced to respond with a discrete project with x number of deliverable outcomes” [1]. It goes on to cite D. Sriskandarajah, the secretary-general of Civicus, a global network of civil society organizations and activists, wrote: ‘We have become a part of the problem rather than the solution… Since demonstrating bang for your buck has become all-important, we divide our work into neat projects, taking on only those endeavours that can produce easily quantifiable outcomes. Reliant on funding to service our own sizeable organizations, we avoid approaches or issues that might threaten our brand or upset our donors. We trade in incremental change’” [1].
We also settle for results while we control them, and don’t ask unpopular questions about who is to sustain these results, with what resources, and for goodness sake, why sustained impact was not funded, designed, implemented and monitored/ evaluated from the very onset in our rush to measurable results?
As this great NGO article by Dinyar Godrej goes on to say, “most media scrutiny of NGO accountability is of how they use funds, their accountability to donors. But what of their accountability towards the recipients of their interventions” [1]? They have no lobbyists to persuade our funders they would like this but nto that, and often such lobbying for their needs falls to the very NGOs that have won these large contracts and tasked with implementing a dizzying array of mandatory input, output, outcome and some impact indicators. We do care deeply about results! US State Department/ USAID has a “Standard Foreign Assistance Master Indicator List” of 2,300 lines in an excel spreadsheet [2]. (There are more indicators still– custom and cross-cutting indicators, the mind boggles).
Wow. But are we asking the right questions? Are we asking what was sustained after all this hard work was done and ended? Rarely. Who should be?! “It is perhaps unrealistic to expect such large structural changes to be delivered by NGOs when governments don’t tackle them either.”
For the rub is this. When we take development over from national governments, largely do not involve country nationals in the funding, design and M&E of projects, then how sustained can these projects still be after we go? Millions are invested, then disappear… Last year, at local debrief at the end of one SEIE Valuing Voices did, the state of affairs became crystal clear when a government official asked us “Can you ever find some funds to fund us to do our own independent evaluations? Even if it is not the projects that they did themselves? We would be happy to get that support…”
When are we no longer responsible for doing great, sustained work? Valuing Voices will let you know what we found regarding the best ways to do SEIEs more. Stay tuned.
What do you think?
P.S. This blog topic prompted me to look for statistics on the number or percentage of funded projects that were renewed. Nothing. Does anyone know how many or what % of projects were extended/ funded again after showing good results? (Often this happens in the form that a successful project in one area of the country gets either funded again or repeated elsewhere in the country or in the world, as have two of our own SEIEs, Niger and Ethiopia). For that matter, what made them so excellent to be replicated? What can we learn?
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).
