by Jindra Cekan | Mar 2, 2017 | Aid effectiveness, Better Evaluation, Corporate Social Responsibility, Dare to Lead, Effective Philanthropy, ex-post evaluation, Foundations, impact evaluation, Impact Investors, international development, Participants, partners, post-project evaluation, SEIE, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Valuing Voices
Leading in Challenging Times:
Sustained and Emerging Impacts Evaluation (SEIEs)
Some American organizations are retrenching, focusing more attention on domestic rather than international programming. Some are pulling back from critique of international development to informing legislators of its benefits; the Center for Global Development’s changed ‘Rethinking US Development Policy’ blog to only “US Development Policy“. UN’s Refugee Agency questions whether to challenge Washington’s tough line on refugees from countries such as Syria, or should it stay quiet in the hopes of protecting its funding [1]?”
Reticence is understandable in this ‘climate’, so to speak, but fear does not change the world, leadership does. Envisioning and creating the world we want gets us there.
There may be no better time to build the evidence base on what works in sustainable development as these are low cost investments if we use national staff and focus research well. We have seen this in the fewer than 1% of all projects that have been evaluated post-closeout for sustainability [2]. At the very least, we can learn what we should do differently in the next design, to fully foster sustainability, once more funding emerges. Many are interested in great results. Hundreds of ‘impact evaluations’ are happening on aid effectiveness; our industry wants to learn what works and what we could do better.
Our SEIE work goes beyond current understanding of ‘impact’ to see what projects our partners and participants can self-sustain ex-post for years to come which is an excellent investment in proving cost-effectiveness. While some governments’ investments can diminish in the short term, national governments, and other funders such as a range of international bilateral and multilateral donors, foundations corporate social responsibility and impact investors do want to invest in provably “sustainable” development [3].
Why should we invest in SEIEs?
- Hundreds of thousands of projects are still being implemented.
- Millions of participants are still hoping what we are doing together will be sustainable.
- Billions of dollars, euros, kwacha, pesos, rupees are being spent on new projects that need to be designed and implemented for future sustainability.
Implementing organizations could be fearful to see what remains once funding and technical assistance are withdrawn, but such a view not only robs our industry of exciting lessons on what did change and was so valued that it was sustained, but also what to not do again. Not returning post-project also short-changes our participants. In our SEIEs, we have found participants and partners creating new ways to carry on, innovating beyond what we could imagine during our assistance.
We also need to start now to design and implement for sustainability. doing SEIEs, we can start to understand the ‘drivers’ behind the Sustainable Development Goals (SDG) results with countries tracking some 120 indicators across 17 goals. Currently countries are tracking up to 230 indicators across the 17 Goals [4]. But while such monitoring shows ‘GDP has increased or ‘under-nourishment has decreased’, there is little or no information on what has caused it. Yet doing and SEIE on a large donor-funded programme, we can explore what elements made projects sustainable and how to do more (or less) there and elsewhere. Such sentinel site support for learning about sustained and emerging impacts is key to understand some of the why, for example, did income or health improve.
Dare to lead, especially in these challenging times. We know of organizations that are doing these evaluations internally, others are publishing them on their sites. Leadership happens at all levels, from internal, technical to managerial and administrative work to external evaluators and consultants as well as public pressure.
How can you foster sustained impact?
- You can advocate for such evaluations
- You can share the SEIE guidance, below, and start to design and implement, monitor and evaluate sustainably in all projects/ proposals you are designing now.
- You can see if your organization has done any post-project sustainability evaluations and we can post them on Valuing Voices’ repository, celebrating your organization.
We can help you learn how to do these. Our partner, Better Evaluation, just published our Sustained and Emerging Impacts Evaluation as a ‘new’ evaluation ‘theme.’
Guidance there shows you [5]:
1. What is SEIE?
2. Why do SEIE?
3. When to do SEIE?
4. Who should be engaged in the evaluation process?
5. What definitions and methods can be used to do an SEIE?
Resources
References
SEIEs will grow as will examples, discussions, and joy as embracing sustainability sprouts, and sends us progressing in yet-unforeseen ways! We are excited to be in the final stages of receiving a research grant to further guide SEIEs. We will share that news in our next blog.
We want to learn from you:
- What do you think needs to be in place for funders to move beyond the funding cycle and do an SEIE?
- What would help to make this type of evaluation more widely undertaken?
- If you have done a post-project evaluation, how did you do it? What were some of the barriers you faced and resources you were able to draw on to overcome them?
How can we lead together to Value the Voices of those we serve!?
(Reposted from https://medium.com/@WhatWeValue/leading-in-challenging-times-sustained-and-emerging-impacts-evaluation-seies-617b33bf4d27#.ec7fcg4ty)
Sources:
[1] Foulkes, I. (2017, February 27). Is there a US diplomacy vacuum at the UN in Geneva? Retrieved from https://www.bbc.com/news/world-europe-39080204
[2] Cekan, J. (2015, March 13). When Funders Move On. Retrieved from https://ssir.org/articles/entry/when_funders_move_on
[3] UN DESA. (2011, March 2). Lasting impact of sustainable development. Retrieved from https://www.un.org/en/development/desa/news/sustainable/sustainable-development.html
[4] UN Statistics Division. SDG Indicators: Global indicator framework for the Sustainable Development Goals and targets of the 2030 Agenda for Sustainable Development. Retrieved March, 2017, from https://unstats.un.org/sdgs/indicators/indicators-list/
[5] Cekan, J., Zivetz, L., & P, R. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE
by Jindra Cekan | Dec 1, 2014 | Beneficiaries, Big data, Evaluation, Evidence-based policy, Foundations, Participants, Philanthropy, Results, Sustainable development
Times are a Changin' in those who Fund Listening… then Doing
So you've been helped by an organization. You think it has a good mission and have actively participated in its activities yet one day (somewhat arbitrarily in your view), it takes you off its list, shuts its doors and moves to another state. What would you feel? Angry? Perplexed? Disappointed?
So a year or two goes by and you get a knock on your door from a similar organization, wanting you to participate with them, that their mission is great, that you will benefit a lot. While you may really want their help, you are understandably wary and wonder if the same will happen. Heck, the last ones didn't tell you why they left, even with unfinished work, nor came back to see how you were faring…
Maybe that won’t happen anymore. Until recently many of our international development participants (some call them beneficiaries) could feel the same way. Our projects came (and went) with set goals, on fixed funding cycles, with little ongoing input from them to influence how projects accomplishes good things, much less learned what happened after projects ended. Rarely have we put into place participant monitoring systems with feedback loops much less listen to participants on how to design for self-sustainability.…
But times are a changin'; there is much to celebrate among funders and implementers, programming and policy makers.
1) There is a happy blizzard of interest in listening to our participants. From Feedback Labs "committed to making governments, NGOs and donors more responsive to the needs of their constituents" and Rita Allen Foundation funding for the Center for Effective Philanthropy's "Hearing from those we seek to Help" to now the Rockefeller and Hewlett Foundation's Effective Philanthropy's beginning a joint Fund for Shared Insight which "provides grants to nonprofit organizations to encourage and incorporate feedback from the people we seek to help; understand the connection between feedback and better results…".
Independent voices abound that are advocating for participants' voices to be heard in design, implementation, monitoring and evaluation: "While we may have a glut of information and even the best of intentions, our initiatives will continue to fall short until we recognize that our ‘beneficiaries’ are really the people who have the solutions that both they and we need." And others call for even more than recognition – participation of the funders in discussions with participants: A recent study by the Center for Effective Philanthropy heard from recipient NGOs that the "funders who best understand our beneficiaries’ needs are the ones who visit us during our programs, meet [those[ served by our organization, spend time talking to them and being with them.”
2) Information and Communication Technologies for Development) has created options of listening to our project participants, learning from them/ with them through mobiles, tablets and other mechanisms (e.g. Catholic Relief Services' ICT4D 6th annual conference with presentations from donors and government as well, Ushahidi which we've celebrated before); IATI, the International Aid Transparency Initiative has spent 6 years fostering sustainable and foreign aid transparent development, now reaching 24 signatory countries, and 290 organizations. A data revolution is taking shape to join donor data, national government statistical data and civil society socio-economic data. There is a brand new initiative at IDS named Doing Development Differently. Listening and learning indeed!
3) And even more importantly, an understanding that development is not a one-size fits all endeavour, is arising. I blogged about Rwanda's success in nutritional impact from allowing communities to address their specific needs and this week New Republic published an excellent article by Michael Hobbes which says "The repeated “success, scale, fail” experience of the last 20 years of development practice suggests something super boring: Development projects thrive or tank according to the specific dynamics of the place in which they’re applied. It’s not that you test something in one place, then scale it up to 50. It’s that you test it in one place, then test it in another, then another." Hobbes goes on to add that what we need is a revision in our expectations of international aid. "The rise of formerly destitute countries into the sweaters-and-smartphones bracket is less a refutation of the impact of development aid than a reality-check of its scale. In 2013, development aid from all the rich countries combined was $134.8 billion, or about $112 per year for each of the world’s 1.2 billion people living on less than $1.25 per day. Did we really expect an extra hundred bucks a year to pull anyone, much less a billion of them, out of poverty?… Even the most wildly successful projects decrease maternal mortality by a few percent here, add an extra year or two of life expectancy there. This isn’t a criticism of the projects themselves. This is how social policy works, in baby steps and trial-and-error and tweaks, not in game changers."
4) What does change the game in the view of Valuing Voices is who we listen to and what we do, for how long. Often, project participants have been the implementers of our solutions rather than the drivers of their own ideas of development; much is lost in translation. As Linda Raftree reports from one Finnish Slush attendee, "“When you think ‘since people are poor, they have nothing, they will really want this thing I’m going to give them,’ you will fail:…“People everywhere already have values, knowledge, relationships, things that they themselves value. This all impacts on what they want and what they are willing to receive. The biggest mistake is assuming that you know what is best, and thinking ‘these people would think like me if I were them.’ That is never the case.” Hallelujah.
Let's listen before we implement our best answers, adapt to specific communities, think of how to foster self-sustainability rather than just successful impact, ask what that is in their terms. Let’s return to listen to participants’ views on sustained impact, on unexpected results… let’s fund this and do development differently!
So how are you listening to participants today?
