by Jindra Cekan | Mar 2, 2017 | Aid effectiveness, Better Evaluation, Corporate Social Responsibility, Dare to Lead, Effective Philanthropy, ex-post evaluation, Foundations, impact evaluation, Impact Investors, international development, Participants, partners, post-project evaluation, SEIE, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Valuing Voices
Leading in Challenging Times:
Sustained and Emerging Impacts Evaluation (SEIEs)
Some American organizations are retrenching, focusing more attention on domestic rather than international programming. Some are pulling back from critique of international development to informing legislators of its benefits; the Center for Global Development’s changed ‘Rethinking US Development Policy’ blog to only “US Development Policy“. UN’s Refugee Agency questions whether to challenge Washington’s tough line on refugees from countries such as Syria, or should it stay quiet in the hopes of protecting its funding [1]?”
Reticence is understandable in this ‘climate’, so to speak, but fear does not change the world, leadership does. Envisioning and creating the world we want gets us there.
There may be no better time to build the evidence base on what works in sustainable development as these are low cost investments if we use national staff and focus research well. We have seen this in the fewer than 1% of all projects that have been evaluated post-closeout for sustainability [2]. At the very least, we can learn what we should do differently in the next design, to fully foster sustainability, once more funding emerges. Many are interested in great results. Hundreds of ‘impact evaluations’ are happening on aid effectiveness; our industry wants to learn what works and what we could do better.
Our SEIE work goes beyond current understanding of ‘impact’ to see what projects our partners and participants can self-sustain ex-post for years to come which is an excellent investment in proving cost-effectiveness. While some governments’ investments can diminish in the short term, national governments, and other funders such as a range of international bilateral and multilateral donors, foundations corporate social responsibility and impact investors do want to invest in provably “sustainable” development [3].
Why should we invest in SEIEs?
- Hundreds of thousands of projects are still being implemented.
- Millions of participants are still hoping what we are doing together will be sustainable.
- Billions of dollars, euros, kwacha, pesos, rupees are being spent on new projects that need to be designed and implemented for future sustainability.
Implementing organizations could be fearful to see what remains once funding and technical assistance are withdrawn, but such a view not only robs our industry of exciting lessons on what did change and was so valued that it was sustained, but also what to not do again. Not returning post-project also short-changes our participants. In our SEIEs, we have found participants and partners creating new ways to carry on, innovating beyond what we could imagine during our assistance.
We also need to start now to design and implement for sustainability. doing SEIEs, we can start to understand the ‘drivers’ behind the Sustainable Development Goals (SDG) results with countries tracking some 120 indicators across 17 goals. Currently countries are tracking up to 230 indicators across the 17 Goals [4]. But while such monitoring shows ‘GDP has increased or ‘under-nourishment has decreased’, there is little or no information on what has caused it. Yet doing and SEIE on a large donor-funded programme, we can explore what elements made projects sustainable and how to do more (or less) there and elsewhere. Such sentinel site support for learning about sustained and emerging impacts is key to understand some of the why, for example, did income or health improve.
Dare to lead, especially in these challenging times. We know of organizations that are doing these evaluations internally, others are publishing them on their sites. Leadership happens at all levels, from internal, technical to managerial and administrative work to external evaluators and consultants as well as public pressure.
How can you foster sustained impact?
- You can advocate for such evaluations
- You can share the SEIE guidance, below, and start to design and implement, monitor and evaluate sustainably in all projects/ proposals you are designing now.
- You can see if your organization has done any post-project sustainability evaluations and we can post them on Valuing Voices’ repository, celebrating your organization.
We can help you learn how to do these. Our partner, Better Evaluation, just published our Sustained and Emerging Impacts Evaluation as a ‘new’ evaluation ‘theme.’
Guidance there shows you [5]:
1. What is SEIE?
2. Why do SEIE?
3. When to do SEIE?
4. Who should be engaged in the evaluation process?
5. What definitions and methods can be used to do an SEIE?
Resources
References
SEIEs will grow as will examples, discussions, and joy as embracing sustainability sprouts, and sends us progressing in yet-unforeseen ways! We are excited to be in the final stages of receiving a research grant to further guide SEIEs. We will share that news in our next blog.
We want to learn from you:
- What do you think needs to be in place for funders to move beyond the funding cycle and do an SEIE?
- What would help to make this type of evaluation more widely undertaken?
- If you have done a post-project evaluation, how did you do it? What were some of the barriers you faced and resources you were able to draw on to overcome them?
How can we lead together to Value the Voices of those we serve!?
(Reposted from https://medium.com/@WhatWeValue/leading-in-challenging-times-sustained-and-emerging-impacts-evaluation-seies-617b33bf4d27#.ec7fcg4ty)
Sources:
[1] Foulkes, I. (2017, February 27). Is there a US diplomacy vacuum at the UN in Geneva? Retrieved from https://www.bbc.com/news/world-europe-39080204
[2] Cekan, J. (2015, March 13). When Funders Move On. Retrieved from https://ssir.org/articles/entry/when_funders_move_on
[3] UN DESA. (2011, March 2). Lasting impact of sustainable development. Retrieved from https://www.un.org/en/development/desa/news/sustainable/sustainable-development.html
[4] UN Statistics Division. SDG Indicators: Global indicator framework for the Sustainable Development Goals and targets of the 2030 Agenda for Sustainable Development. Retrieved March, 2017, from https://unstats.un.org/sdgs/indicators/indicators-list/
[5] Cekan, J., Zivetz, L., & P, R. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE
by Jindra Cekan | Sep 16, 2015 | Accountability, Aid effectiveness, Community Driven Development, Effective Philanthropy, Failure, Global Giving, International aid, Participants, Sustainability
Face our fears! Learn from failure…
Global Giving has a nice example about getting participant feedback success/ failrure of a project their fundraising funded. The organization failed which was sorrowful to the players in West Africa and funders worldwide (I too am a soccer-mom). Yes, it failed. It is hard to read those words and all involved faced it and they learned from it. Thanks to a sprited discussion on Pelican Platform for Evidence-based Learning about a plethora of unintended impacts, I learned about the organization committed to learning from failures: Admitting Failure.
Maybe some of you have heard of FailFests, such as the one in Raleigh N.C US this year that answered "Why are we doing this?" with "We’re on a mission to erase the stigma around failure. The more we talk about it, the more we can learn from it. Failure doesn’t have to be fatal!" You may know Engineers without Borders Canada has been an early proponent of "openly acknowledging failure is often a catalyst for innovation that takes our work from good to great." Such examples are heartening, especially for participatory sustainability evaluation.
In 2014 a met a fellow evaluator at the American Evaluation Association Conference where I was presenting on Post-Project Sustainability Evaluation. They told me their organization was waiting for a really successful project to end so they could evaluate its sustainability. Another evaluator told me of a post-project evaluation that was hidden away, unpublished, after findings were negative. And that's where the problem lies: our international development industry's neurosis about presenting anything unsuccessful.
No wonder so few projects have been evaluated post-project, as we fear:
* What if activities and outcomes aren't sustained (note: we didn't design them that way, often we opt for quicker wins achievable only with large resource inflows of funds and technical help)?
* What if our funders find out that funds had limited impact or partners had no means to continue good programming?
* What if we had entirely unintended impacts that favored some over others (well beyond our expected logical frameworks and Theories of Change)?
Hallelujah 🙂 Why can't we see this as good news that propels us to improve our projects by listening to what worked? To learn not to do what didn't work again?! To design in locally sustainable ways now that we have learned?
* What if we learn that our resources and empowerment led them to succeed on their own terms in ways we couldn't imagine, far exceeding the planned impacts we had expected?
* What if we find that unexpected outcomes showcased ways groups within communities stood on their feet, making development work for them on their terms?
Would we design and implement, monitor and evaluate projects differently? We'll see. Organizations such as USAID’s Food for Peace funded a four-country study on exit-strategies (forthcoming 2015), are looking at success and failure. Catholic Relief Services hired Valuing Voices to do sustainability evaluation in Africa. Others may follow…
Post-project sustainability evaluations expose us to learning the full range. Let's be brave and ask, learn and innovate, making aid and philanthropy more effective, and learn from failure for success!
by Jindra Cekan | Aug 28, 2015 | Accountability, Aid effectiveness, Beneficiaries, Center for Global Development, Charity Navigator, Effective Philanthropy, Evaluation, Hewlett Foundation, International aid, Making All Voices Count, OECD, ONE, OXFAM, post-project evaluation, self-sustainability, stakeholders, Sustainable development, USAID
Altruistic Accountability… for Sustainability
Many of us in international development feel a sense of responsibility for others to be well, and for our work to improve their lives as well as for the work to be done in good stewardship of aid resources and optimizing their impact. As Matthieu Ricard writes, "Altruism is a benevolent state of mind. To be altruistic is to be concerned about the fate of all those around us and to wish them well. This should be done together with the determination to act for their benefit. Valuing others is the main state of mind that leads to altruism." We also feel a responsibility to our international aid donors and taxpayers. We who implement, monitor and evaluate projects work to ensure that the altruism of aid is responsible to both donors and recipients.
Altruism appears most vividly when implementers issue appeals after disasters, with millions donated as a result, but unsung heroes are also development workers. Organizations such as Charity Navigator, ONE and Center for Global Development on how well US organizations spent funds and track donor-country policy accountability. Thoughtful donor studies such as French Development Agency’s OECD study report on the power of AidWatch and Reality of Aid intiatives in Europe for their taxpayers.
But who is pushing for our donor’s accountability to the country’s participants themselves? While USAID funds many program evaluations, some of which “identify promising, effective…strategies and to conduct needs assessments and other research to guide program decisions”, they are always at project end, rather than looking at sustainability of the outcomes and impacts, and focus on Congressional and domestic listeners. This is no funding and no small audience. The US Department of State/ USAID’s FY13 Summary report states that in fiscal year 2013, USAID had $23.8 billion to disburse, over $12 billion for programming. While total beneficiary (participant) numbers were not provided, emergency food assistance alone used $981 million for nearly 21.6 million people in 25 countries.
So who is a watchdog for what results? OXFAM may excellently highlight opportunities for better programming. 3ie does many studies looking at projected impact and does systemic reviews (but only three were post-project). Challenges such as Making All Voices Count may fund channels for country-nationals to hold their own governments responsible, but can in-country project participants ever demand sustainable results from anyone but their own governments? Herein lies the crux of the issue. Unless governments demand it (unlikely in ‘free’ aid), only pressure from donor country nationals (you? we) can push for changes.
At the core of Valuing Voices mission is advocacy for altruistic accountability of the sustainability of projects to country-ownership at all levels. For us, this involves valuing and also giving voice to those supported by, and also tasked with doing ‘sustainable projects’. Unless we know how sustainable our development projects have been, we have only temporarily helped those in greatest need. This means looking beyond whether funding continued to whether the benefits of an activity or even the existence of entire local NGOs tasked with this actually continued after funding was withdrawn. Unless we strive to learn what has continued to work best or failed to be continued after projects left in the views of the participants themselves, we can let down the very people who have entrusted us with hopes of a self-sustainable future of well being. Unless we listen to project staff and local partners to see what program staff felt they did right/wrong, and national partners felt they were supported to do keep doing right, we have minimized success of future projects. While increasing numbers of organizations such as Hewlett Foundation fund work to “increase the responsiveness of governments to their citizens’ needs. We do this by working to make governments more transparent and accountable,” the long-term effectiveness of our donor development assistance is not yet visible.
OECD guidelines on corporate accountability and transparency are illuminating. Adapting it from State-Corporate to Non-profit-State is interesting. For how well have we considered who ‘owns’ these development projects in practical terms from inception onward? Our donors? Implementing agencies? Local partners and communities?
OECD Guidelines on Corporate Governance of State-Owned Enterprises
1: The State Acting as an Owner
2: Equitable Treatment and Relations with Shareholders
3: Ensuring an Effective Legal and Regulatory Framework for State-Owned Enterprises
4: Transparency and Disclosure
How well do we design projects along these lines to do this successfully? Not terrifically:
-
Too often ‘stakeholders’ are not consulted at the very inception of the proposal design, only at design or implementation
-
Too often our work is aimed at making only our ‘client’- our donors- happy with our results rather than the country nationals who are tasked with self-sustaining them.
-
Too often handover is done at the 11th hour, not transferring it throughout implementation or building local capacity for those taking over be true projects’ owners.
But it is coming, through changing societal trends. On the data-access front, USAID (and differently, other European donors) have promised to modernize diplomacy and development by 2017 by “increas[ing] the number and effectiveness of communication and collaboration tools that leverage interactive digital platforms to improve direct engagement with both domestic and foreign publics. This will include increasing the number of publicly available data sets and ensuring that USAID-funded evaluations are published online, expanding publicly available foreign assistance data, increasing the number of repeat users of International Information.” Now to generate and add self- sustainability data to inform future projects!
Second, on the they-are-se front, our basic human nature, according to Ricard, lends itself to altruism. “Let's assume that the majority of us are basically good people who are willing to build a better world. In that case, we can do so together thanks to altruism. If we have more consideration for others, we will promote a more caring economy, and we will promote harmony in society and remedy inequalities.” Let’s get going…
by valuingvoicesjin | Dec 11, 2014 | Africa, Aid effectiveness, Big data, Effective Philanthropy, Evaluation, Feedback loops, foreign aid, IATI, International aid, International non-profits, Local Participants, Milenium Development Goals, Open Data, Participants, self-sustainability, Sustainability, water
What should projects accomplish… and for whom?
An unnamed international non-profit client contacted me to evaluate their resilience project mid-stream, to gauge prospects for sustainable handover. EUREKA, I thought! After email discussions with them I drafted an evaluation process that included learning from a variety of stakeholders, ranging from Ministries, local government and the national University who were to take over the programming work about what they thought would be most sustainable once the project ended and how in the next two years the project could best foster self-sustainability by country-nationals. I projected several weeks for in-depth participatory discussions with local youth groups and sentinel communities directly affected by the food security/ climate change onslaught and who benefited from resilience activities to learn what had worked, what didn’t and who would take what self-responsibility locally going forward.
Pleased with myself, I sent off a detailed proposal. The non-profit soon answered that I hadn’t fully understood my task. In their view the main task at hand was to determine what the country needed the non-profit to keep doing, so the donor could be convinced to extend their (U.S.-based) funding. The question at hand became how could I change my evaluation to feed them back this key information for the next proposal design?
Maybe it was me, maybe it was the autumn winds, maybe it was my inability to sufficiently subsume long-term sustainability questions under shorter-term non-profit financing interests that led me to drop this. Maybe the elephant in the living room that is often unspoken is the need for some non-profits to prioritize their own organizational sustainability to ‘do good’ via donor funding rather than working for community self-sustainability.
Maybe donor/funders should share this blame, needing to push funding out, proving success at any cost to get more funding and so the cycle goes on. As a Feedback Lab feature on a Effective Philanthropy report recently stated: “Only rarely do funders ask, ‘What do the people you are trying to help actually think about what you are doing?’ Participants in the CEP study say that funders rarely provide the resources to find the answer. Nor do funders seem to care whether or not grantees are changing behavior and programs in response to how the ultimate beneficiaries respond” [1].
And how much responsibility do communities themselves hold for not balking? Why are they so often ‘price-takers’ (in economic terms) rather than ‘price-makers’? As wise Judi Aubel asked in a recent evaluation list-serve discussion “When will communities rise up to demand that the “development” resources designed to support/strengthen them be spent on programs/strategies which correspond to their concerns/priorities??”
We can help them do just that by creating good conditions for them to be heard. We can push advocates to work to ensure the incoming Sustainable Development Goals (post-MDGs) listen to what recipient nations feel are sustainable, more than funders. We can help their voices be heard via systems that enable donor/ implementers to learn from citizen feedback, such as Keystone has via their Constituent Voice practice (in January 2015 it is launching an online feedback data sharing platform called the Feedback Commons) or GlobalGiving’s new Effectiveness Dashboard (see Feedback Labs).
We can do it locally in our work in the field, shifting the focus from our expertise to theirs, from our powerfulness to theirs. In field evaluations can use Empowerment Evaluation. We can fund feedback loops pre-RFP (requests for proposals), during project design, implementation and beyond, with the right incentives tools for learning from community and local and national-level input so that country-led development begins to be actual not just a nice platitude. We can fund ValuingVoices’ self-sustainability research on what lasts after projects end. We can conserve project content and data in Open Data formats for long-term learning from country-nationals.
Most of all, we can honour our participants as experts, which is what I strive to do in my work. I’ll leave you with a story from Mali. in 1991 I was doing famine-prevention research in Koulikoro Mali where average rainfall is 100mm a year (4 inches). I accompanied women I was interviewing to a deep well which was 100m deep (300 feet). They used plastic pliable buckets and the first five drew up 90% of the bucket full. When I asked to try, they seriously gave me a bucket. I laughed, as did they when we saw that only 20% of my bucket was full. I had splashed the other 80% out on the way up. Who’s the expert?
How are we helping them get more of what they need, rather than what we are willing to give? How are we prioritizing their needs over our organizational income? How are we #ValuingVoices?
Sources:
[1] The Center for Effective Philanthropy. (2014, October 27). Closing the Citizen Feedback Loop. Retrieved December 2014, from https://web.archive.org/web/20141031130101/https://feedbacklabs.org/closing-the-citizen-feedback-loop/
[2] Better Evaluation. (n.d.). Empowerment Evaluation. Retrieved December 2014, from https://www.betterevaluation.org/plan/approach/empowerment_evaluation
[3] Sonjara. (2016). Content and Data: Intangible Assets Part V. Retrieved from http://www.sonjara.com/blog?article_id=135
