Grow the .002% of all global development projects that are evaluated ex-post closure for sustainability

Grow the .002% of all global development projects that are evaluated ex-post closure for sustainability

It seems like ‘fake news’ that after decades of global development so few evaluations would have peered back in time to see what was sustained. While I was consulting to the Policy Planning and Learning Bureau at USAID, I asked the head of this M&E department who does ex-post sustainability evaluation as I knew USAID had done some in the 1980s, Cindy Clapp-Wincek answered ‘No one, there are no incentives to do it.’ (She later became our advisor.)

Disbelieving, I did a year of secondary keyword research before devoting my professional consulting life to advocating for and doing ex-post evaluations of sustained outcomes and impacts. I searched USAID, OECD, and other bilateral and later multilateral donors’ databases and found thousands of studies, most of which were inaccurately named ‘ex-post’ or ‘post-closure’ studies.  Some of the roughly 1,000 projects I looked at at USAID and OECD that came up under ‘ex-post’, ‘ex post’, ‘post closure’ were final evaluations that were slightly delayed, a few were evaluations that were at least one year after closure, but were desk studies without interviews. Surprisingly, the vast majority of final evaluations found were those that only recommended ex-post evaluation several years later to confirm projected sustainability.

 

 

 

 

 

 

 

 

In 2016 at the American Evaluation Association conference, a group of us did a presentation. In it, I cited these statistics from of 1st year of Valuing Voices’ research:

  • Of 900+ “ex-post” “ex post” “post closure” documents in USAID’s DEC database, there were only 12 actual post-project evaluations with fieldwork have been done in the last 20 years
  • Of 12,000 World Bank projects – only 33 post-project evaluations asked ‘stakeholders’ for input, and only 3 showed clearly they talked to participants
  • In 2010 Asian Development Bank conducted 491 desk reviews of completed projects, and returned to 18 actual field-based post-project evaluations that included participant voices; they have done only this 1 study.
  • We found no evaluations by recipient governments of aid projects’ sustainability

12 years of research, advocacy and fieldwork later, the ‘catalysts’ database on Valuing Voices now shows actual fieldwork-informed evaluations by 40 organizations that had actual ex-posts that returned to the field to ask participants and project partners what was sustained, highlighting 92 ex-posts.

How many ex-post project closure evaluations have been done? .002% of all projects. The 0.002% statistic looks at just public foreign development aid from 1960 (not even counting private funding such as foundations or gifts to organizations, which isn’t tracked in any publicly available database). Calculating aggregated OECD aid statistics (excluding private because it’s only recent data) over 62 years $5.6 trillion by 2022 (thanks to Rebecca Regan-Sachs for the updated #s).

I then estimated 3.000 actual ex-posts which comes from 2,500 JICA projects plus almost 500 other projects that I have either found looking through databases all across the spectrum from governments and multilaterals (almost 100 in our catalysts, and am assuming there must be 400 others done in the 1980s-2000 like USAID and the World Bank).

Without a huge research team it is improssible to aggregate data on the total number of projects by all donors. So I extrapolated from project activity disbursements of one year (2022) for Mali on the www.foreignassistance.gov page. In my 35 years of experience, Mali, where I did my doctoral research, typifies he average USAID aid recipient. They had 382 projects going in 2022. I rounded up to 400 projects x 70 years (since 1960 when OECD data began) x 100 countries by just one donor (of the 150 possible recipient countries, to be conservative). This comes to 2.8 million projects. So if we take 39 OECD countries as donors (given most have far less to give than US), in total 109 million publicly funded aid projects disbursed $5.6 trillion since 1960. While final evaluations are industry standard, only .002% is the estimated number of ex-post evaluations of projects the were evaluated with data from local participants and partners of the 109 million projects .

This became Valuing Voices focus, and we created an open-access database for learning, and conducted our own  My team and I identified 92 ex-posts that returned to ask locals what lasted, what didn’t, why, and what emerged from their own efforts. We also created evaluability checklists and created a new evaluation, Sustained and Emerging Impacts Evaluation that included examining not just what donors put in place to last, but also what emerged outcomes from local efforts to sustain results with more limited resources, partnerships, capacities and local ownership/motivation. These four drivers were found by Rogers and Coates for USAID’s food security exit study in 2015). We have done 15 ex-posts for 9 clients since 2006 and shared Adaptation Fund ex-post training materials in 2023.

 

Yet the public assumes we know our development is sustainable. 2015’s ‘Sustainable Development Goals‘ focused aid on 17 themes, which was to generate $12 trillion more in annual spending on SDG sectors than the;$21 trillion already being invested each year. Nonetheless, a recent UN report states that there is now a $4 trillion annual financing gap to achieve the SDGs. All this funding goes to projects that are currently implemented, not to evaluate what had been sustained from past projects that already closed. Such learning from what succeeded or failed, or what emerged from local efforts to keep activities and results going is pivotal to improving current and future programming is almost wholly missing from the dialogue; I know, I asked multiple SDG evaluation experts.

 

Why do we return to learn so rarely? There are many reasons, the most prosaic among them being administrative.

  • When aid funds are spent over 2-10 years, projects are closed, evaluated at the end, ‘handed over’ to national governments, and no additional funding exists to return ‘ex-post’ closure to learn.
  • Next is the push to continue to improve lives through implementation which means low rates of overhead allocated to M&E and learning during, much less after closure.
  • Another is the assumption that ‘old’ projects differ so much from new ones, but there are few differences. After all there are only so many ways to grow food, feed the malnourished, educate children; evaluating ‘old’ projects can teach ‘new’ projects.
  • A last major one, from Valuing Voices’ research of 12 years may be the largest: Fear of admitting failure. Please read Valuing Voices’ 2016 blog highlighted many Lessons about Funding, Assumptions and Fears (Part 3). One US aid lobbyist told me in 2017 that I must not share this lack of learning about sustained impacts because it could imperil US aid funding; I told her I had to tell people because lives were at stake.
  • Overall, there is much to learn; most ex-post evaluations show mixed results. None show 100% sustainability and while most show 30-60% sustainability, none are 0% sustained either. If we don’t learn to replicate what worked and cease what didn’t now, then future programming will be as flawed and successes, especially brilliant emerging locally designed ex-post outcomes such as Niger’s local funding of redesign of health incentives will remain hidden.

 

Occasionally donors invest in sets of ex-post learning evaluations such as USAID’s ‘global waters’ seven water/ sanitation evaluations linked to the E3 Bureau taking sustainability as a strategic goal. Yet the overall findings from USAID’s own staff of these ex-posts Drivers of WASH study were chilling. While 25 million gained access to drinking water and 18 million to basic sanitation, ‘they have largely not endured.’ But the good news in such research is that the donor learned that infrastructure fails when spare parts are not accessible and maintenance not funded or performed, which can be planned for and addressed during implementation by investing in resources and partnerships. They learned that relying on volunteers is unreliable and management needs to be bolstered, which can lead to some implementation funding to be focused on capacities and local ownership. We can plan better for sustainability by learning from ex-post and exit studies (see Valuing Voices’ checklists in this 2023 article on Fostering Values-Driven Sustainability).

 

And since 2019, three climate funds, the Adaptation Fund, the Global Environmental Facility, and the Climate Investment Funds have turned to ex-post evaluations to look at sustainability and longer-term resilience and even transformation, given environmental shocks may take years to affect the project sites. The Adaptation Fund has done four ex-posts, with more to come in 2024/25, and the CIF is beginning now. The GEF has done a Post-Completion Assessment Pilot for the Yellow Sea Region . Hopeful!

Follow our path on Ex-Post Sustainability and Resilience Seasonal Joy

Follow our path on Ex-Post Sustainability and Resilience Seasonal Joy

Rarely do we get to teach, innovate, learn, and expand a hidden corner of one’s field. This is what is I am experiencing, and the Fund is sharing. Here is the Adaptation Fund’s ex-post project evaluation of sustainability and resilience path and progress in 2021.

 

I am quoting liberally and highlighting our work from the Adaptation Fund’s website where their commitment to learning from what lasts is clear. “Ex post evaluations are a key element of the AF-TERG FY21-FY23 strategy and work programme, originating from the request of the Adaptation Fund Board to develop post-implementation learning for Fund projects and programmes and provide accountability of results financed by the Fund. They intend to evaluate aspects of both sustainability of outcomes and climate resilience, and over time feed into ex-post-evaluation-informed adjustments within the Fund’s Monitoring Evaluation and Learning (MEL) processes.”

 

How are we defining sustainability’s path to evaluate it? Here is a flowchart from our training:

 

There are four phases from 0 to 3:
Phase 0 Foundational Review: Not only was this work preceded by months of background research on both evaluability of their young portfolio (e.g., under 20 of the 100 projects funded were closed at least three years, a selection criteria we had) and secondary research on evidence of ex-post sustainability evaluation in climate change/ resilience across the Adaptation Fund’s sectors.

Phase 1 Framework and Pilots Shortlist: Our Phase 1 report from mid-2021 provided an overview of the first stage of ex-post evaluations, outlining methods and identifying a list of potential projects for ex-post evaluation pilots from the Fund’s 17 completed, evaluated projects. The framework presented in the report introduced possible methods to evaluate the sustainability of project outcomes, considering the characteristics, strengths, and weaknesses of the Fund portfolio. It also presents an analysis tool to assess climate resilience, bearing in mind that this area is pivotal to climate change adaptation yet has rarely been measured.

 

 

 

 

 

 

 

 

The Phase one report on ex post project sustainability evaluation

Vetting and pilot selection, revised design for evaluating sustained outcomes related to resilience to climate change. Key aspects are: 1) Timing (3-5 years since closure or projects at least 4 years long within the last 5 years and seasonality matches the final evaluation) and 2) Good quality of implementation and M&E with measurable outputs and outcomes traceable to impact(s) and 3) Safety to do fieldwork re: Covid, civil peace, etc.

 

 

We (my so-clever colleagues Meg Spearman and Dennis Bours) introduced a new resilience analysis tool that includes consideration of the climate disturbances, the human and natural systems (and their nexus) affected by and affecting project outcomes. This includes five characteristics of resilience in the outcomes (presence of feedback loops, at scale, plus being diverse, dynamic, and redundant) and means/actions to support outcomes. Resilience can be identified via a clear summary of the structures (S) and functions (F) that typify Resistance, Resilience and Transformation showing where a project is and is moving towards. It is a typology of resistance-resilience-transformation (RRT) onto which the overall project can be mapped based on how actions are designed to maintain or change existing structures and functions. That was integrated into the Adaptation Fund resilience evaluation approach.

 

 

Phase 2 Methods Testing and Ex-post Field-testing: Training of national evaluators and piloting two ex-post evaluations per year includes selecting among these methods to evaluate sustainability ex-post plus the RRT and resilience measures above. In the first ex-post in Samoa’s “Enhancing Resilience of Samoa’s Coastal Communities to Climate Change” (UNDP) happening December 21, it is through qualitative evaluation of wall-infrastructure. The second, Ecuador’s “Enhancing resilience of communities to the adverse effects of climate change on food security, in Pichincha Province and the Jubones River basin “(WFP) has training completed and fieldwork should be from January 22, likely be of food security assets and methods TBD.

 

Phase 3 Evaluations continue, with MEL Capacity Building: Two more years of ex-post pilot evaluations (2 per year) with lesson informing integration into the MEL of the Adaptation Fund. We are already finding out lessons of rigor, of knowledge management, of unexpected benefits of returning years after closure, including indications of sustainability and resilience of the assets, with much more learning to come.

Innovations include “the relative novelty of climate change adaptation portfolios and the limited body of work on ex post evaluation for adaptation, it presents possible methods that will be piloted in field-tested ex post evaluations in fiscal year 2022 (FY22).” This includes piloting shockingly rare evaluation of oft-promised resilience. In the update to AF’s Board three months ago, it transparently outlined shortlisting of five completed projects as potential candidates for the pilots, of which two projects were selected for ex post evaluations. It outlined our process of co-creating the evaluation with national partners to prioritize their learning needs while building national capacity to assess sustainability and resilience of project outcomes in the field onward.

 

Also, training materials for ex post pilots are being shared to foster country and industry learning, focusing on evaluating projects at ex-post and emerging sustainability and resilience, as well as presenting and adapting methods to country and project realities.

 

 

 

 

 

 

 

The training material for ex-post pilots

 

The training had three sessions (which could not have happened without colleague Caroline’s expertise):

  • Part A: Understanding ex-post & resilience evaluations. Introduce and understand ex-post evaluations of sustainability and resilience, especially in the field of climate change adaptation
  • Part B: Discussing country-specific outcome priorities and co-creating learning with stakeholders. Discuss the project and its data more in-depth to understand and select what outcome(s) will be evaluated at ex-post
  • Part C: Developing country-specific methods and approaches. Discuss range of methods with the national evaluator and M&E experts to best evaluate the selected outcome(s) and impact(s)

 

Overall progress can be seen in the document updating the AF’s board: A progress update on ex post evaluations (AFB/EFC.28/Inf.4)

 

So in 2018 I Wished for Sustained Outcomes to be explored and in 2019 I Wished again for more Ex-post Evaluation than Needles in Haystacks. In  2021 some of these wishes are becoming fulfilled!  Seasonal Happiness for me is learning about resilience to climate change, diminished vulnerability and searching for proof of sustainability and emerging outcomes and impact(s) and I am grateful to the Adaptation Fund for its commitment to sustainability.

Accompany us on this path, cheer us on, and do your own ex-post sustainability and resilience evaluations! Happy holidays from the Czech Republic!

Building the Evidence Base for Post Project Evaluation: A report to the Faster Forward Fund

 

Building the Evidence Base for Post Project Evaluation:
A report to the Faster Forward Fund

 

We are delighted to share Valuing Voices’ report on the value added of post-project evaluation, which compares findings from eight end-of-project and subsequent post project evaluations [1].  Many of you are aware of how rarely post project evaluations are undertaken.  As a result, there is little real evidence about project impact on long-term sustainability. Valuing Voices received a grant from Michael Scriven’s Faster Forward Fund to begin to address this gap.

Our findings show that post project evaluations can contribute to better understanding of sustainability impacts, and reveal unexpected and emerging outcomes years after project close. They also indicate ways in which we can design and implement for sustainability.

Finding suitable projects for this review was difficult because so few post project evaluations are done, fewer are publically available, and fewer still had comparable final evaluations and included local voices.  Agencies that fund post project evaluations offer a range of reasons for doing so: to learn, to promote a success, to inform replication or scale, to provide justification for future funding, to promote accountabilities.  However, many funding agencies consider post project evaluation a luxury or not necessary.  JICA and OECD are notable exceptions in this regard.

Highlights include:

  • The review highlights the range of methods that have been used in post project evaluations, and point to the advantages of planning for sustainability measurement from the outset of the project.
  • The cases reviewed in the study highlight the (sometime dramatic) difference between the anticipated trajectory of a project, what is happening as the project ends, and what actually continued, was adapted, ceased or changed course after close out.
  • Taxonomies, knowledge management about evaluation, data retrieval/ retention, analysis, use and dissemination are elements of sustained impact evaluation that require attention.
  • Little documentation is available about how post project evaluations have actually informed and influenced organizational learning, sectoral dialogue or future programming.
  • Post project evaluations shed particularly interesting light on what emerged post-project that was entirely due to the efforts and resources of participants and partners after project investments stopped. More on these Sustained and Emerging Impacts Evaluations (SEIEs) at Better Evaluation.

As part of this report, Valuing Voices created an evaluability checklist for assessing whether a post project evaluation is viable, as well as a checklist for measuring sustainability starting at the beginning of the project cycle [2].

We welcome your comments on this report and checklists, and encourage you to share it in your networks and get us feedback on their use.  Please use the report and findings to advocate for more post project sustainability impact evaluations which will contribute to greater evidence-based learning about project sustainability.  Valuing Voices is among a handful of organizations who do post-project evaluations and we can either conduct one or refer you to another who does.

 

Thank you,

Laurie Zivetz, MPH, PhD and Jindra Cekan, PhD, with Kate Robins, MPH, PhD of Valuing Voices

 

The full report is available here:

https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf

 

Sources:

[1] Zivetz, L., Cekan, J., & Robbins, K. (2017, May). Building the Evidence Base for Post-Project Evaluation: Case Study Review and Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf

[2] Zivetz, L., & Cekan, J. (n.d.). Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2017/08/Valuing-Voices-Checklists.pdf
 

How ‘new’ are our projects… and who is aiming at the right outcomes?

 

How ‘new’ are our projects… and who is aiming at the right outcomes?

 

Valuing Voices exciting news is we have received research grant funding from the esteemed evaluator, Michael Scriven’s Faster Forward Fund. We’re looking into the value-added of (ex-post) Sustained and Emerging Impacts Evaluations (SEIEs) and we are doing the research now. We will be documenting methods used and discuss how best to evaluate such sustained impacts after project close-out. Very exciting stuff in this staggeringly ‘new’ field of evaluation.

During this research, a senior international development evaluation expert told us that they can’t return to evaluate now-closed projects because they aren’t the same projects anymore (after closeout) and we are no longer responsible for the results. That took my breath away.

All new projects come from old projects… we recycle old project design most of the time, occasionally making substantive changes in targeting or design but much of how we design and implement remains the same. And while we thoroughly evaluate them during implementation, learning ex-post is a key missing link which all projects in the future can benefit from as we do similar interventions and track similar outcomes year after year but we rarely know which ones were sustained or emerged anew. There absolutely are aspects that get adapted but there are only so many ways to heal the sick, improve crop growth, save money, learn to read and so on, and there is a world we need to learn about what enabled some to be sustained and even morph into new results!

This excellent article, Do NGOs (non-governmental organization) help?, notes that “due to donor pressure [NGOs] are increasingly forced to respond with a discrete project with x number of deliverable outcomes” [1]. It goes on to cite D. Sriskandarajah, the secretary-general of Civicus, a global network of civil society organizations and activists, wrote: ‘We have become a part of the problem rather than the solution… Since demonstrating bang for your buck has become all-important, we divide our work into neat projects, taking on only those endeavours that can produce easily quantifiable outcomes. Reliant on funding to service our own sizeable organizations, we avoid approaches or issues that might threaten our brand or upset our donors. We trade in incremental change’” [1].

We also settle for results while we control them, and don’t ask unpopular questions about who is to sustain these results, with what resources, and for goodness sake, why sustained impact was not funded, designed, implemented and monitored/ evaluated from the very onset in our rush to measurable results?

 

DontSettleForLess

 

As this great NGO article by Dinyar Godrej goes on to say, “most media scrutiny of NGO accountability is of how they use funds, their accountability to donors. But what of their accountability towards the recipients of their interventions” [1]? They have no lobbyists to persuade our funders they would like this but nto that, and often such lobbying for their needs falls to the very NGOs that have won these large contracts and tasked with implementing a dizzying array of mandatory input, output, outcome and some impact indicators. We do care deeply about results! US State Department/ USAID has a “Standard Foreign Assistance Master Indicator List” of 2,300 lines in an excel spreadsheet [2]. (There are more indicators still– custom and cross-cutting indicators, the mind boggles).

Wow. But are we asking the right questions? Are we asking what was sustained after all this hard work was done and ended? Rarely. Who should be?! “It is perhaps unrealistic to expect such large structural changes to be delivered by NGOs when governments don’t tackle them either.”

For the rub is this. When we take development over from national governments, largely do not involve country nationals in the funding, design and M&E of projects, then how sustained can these projects still be after we go? Millions are invested, then disappear… Last year, at local debrief at the end of one SEIE Valuing Voices did, the state of affairs became crystal clear when a government official asked us “Can you ever find some funds to fund us to do our own independent evaluations? Even if it is not the projects that they did themselves? We would be happy to get that support…”

When are we no longer responsible for doing great, sustained work? Valuing Voices will let you know what we found regarding the best ways to do SEIEs more. Stay tuned.

What do you think?

P.S. This blog topic prompted me to look for statistics on the number or percentage of funded projects that were renewed. Nothing.  Does anyone know how many or what % of projects were extended/ funded again after showing good results? (Often this happens in the form that a successful project in one area of the country gets either funded again or repeated elsewhere in the country or in the world, as have two of our own SEIEs, Niger and Ethiopia). For that matter, what made them so excellent to be replicated? What can we learn?

 

 

Sources:

[1] Godrej, D. (2014, December 1). NGOs – Do They Help? Retrieved from https://newint.org/features/2014/12/01/ngos-keynote/

[2] US Department of State. (n.d.). Standard Foreign Assistance Indicators. Retrieved 2017, from https://web.archive.org/web/20170404072145/https://www.state.gov/f/indicators/index.htm

 

Leading in Challenging Times: Sustained and Emerging Impacts Evaluation (SEIEs)- reposted from Medium.com

 

Leading in Challenging Times:
Sustained and Emerging Impacts Evaluation (SEIEs)

 

Some American organizations are retrenching, focusing more attention on domestic rather than international programming. Some are pulling back from critique of international development to informing legislators of its benefits; the Center for Global Development’s changed ‘Rethinking US Development Policy’ blog to only “US Development Policy“. UN’s Refugee Agency questions whether to challenge Washington’s tough line on refugees from countries such as Syria, or should it stay quiet in the hopes of protecting its funding [1]?”

Reticence is understandable in this ‘climate’, so to speak, but fear does not change the world, leadership does. Envisioning and creating the world we want gets us there.

There may be no better time to build the evidence base on what works in sustainable development as these are low cost investments if we use national staff and focus research well. We have seen this in the fewer than 1% of all projects that have been evaluated post-closeout for sustainability [2]. At the very least, we can learn what we should do differently in the next design, to fully foster sustainability, once more funding emerges. Many are interested in great results. Hundreds of ‘impact evaluations’ are happening on aid effectiveness; our industry wants to learn what works and what we could do better.

Our SEIE work goes beyond current understanding of ‘impact’ to see what projects our partners and participants can self-sustain ex-post for years to come which is an excellent investment in proving cost-effectiveness. While some governments’ investments can diminish in the short term, national governments, and other funders such as a range of international bilateral and multilateral donors, foundations corporate social responsibility and impact investors do want to invest in provably “sustainable” development [3].

 

Why should we invest in SEIEs?

  • Hundreds of thousands of projects are still being implemented.
  • Millions of participants are still hoping what we are doing together will be sustainable.
  • Billions of dollars, euros, kwacha, pesos, rupees are being spent on new projects that need to be designed and implemented for future sustainability.

 

Implementing organizations could be fearful to see what remains once funding and technical assistance are withdrawn, but such a view not only robs our industry of exciting lessons on what did change and was so valued that it was sustained, but also what to not do again. Not returning post-project also short-changes our participants. In our SEIEs, we have found participants and partners creating new ways to carry on, innovating beyond what we could imagine during our assistance.

We also need to start now to design and implement for sustainability. doing SEIEs, we can start to understand the ‘drivers’ behind the Sustainable Development Goals (SDG) results with countries tracking some 120 indicators across 17 goals. Currently countries are tracking up to 230 indicators across the 17 Goals [4]. But while such monitoring shows ‘GDP has increased or ‘under-nourishment has decreased’, there is little or no information on what has caused it. Yet doing and SEIE on a large donor-funded programme, we can explore what elements made projects sustainable and how to do more (or less) there and elsewhere. Such sentinel site support for learning about sustained and emerging impacts is key to understand some of the why, for example, did income or health improve.

 

 

Dare to lead, especially in these challenging times. We know of organizations that are doing these evaluations internally, others are publishing them on their sites. Leadership happens at all levels, from internal, technical to managerial and administrative work to external evaluators and consultants as well as public pressure.

 

How can you foster sustained impact?

  • You can advocate for such evaluations
  • You can share the SEIE guidance, below, and start to design and implement, monitor and evaluate sustainably in all projects/ proposals you are designing now.
  • You can see if your organization has done any post-project sustainability evaluations and we can post them on Valuing Voices’ repository, celebrating your organization.

 

We can help you learn how to do these. Our partner, Better Evaluation, just published our Sustained and Emerging Impacts Evaluation as a ‘new’ evaluation ‘theme.’

Guidance there shows you [5]:

1. What is SEIE?
2. Why do SEIE?
3. When to do SEIE?
4. Who should be engaged in the evaluation process?
5. What definitions and methods can be used to do an SEIE?
Resources
References

SEIEs will grow as will examples, discussions, and joy as embracing sustainability sprouts, and sends us progressing in yet-unforeseen ways! We are excited to be in the final stages of receiving a research grant to further guide SEIEs. We will share that news in our next blog.

 

We want to learn from you:

  • What do you think needs to be in place for funders to move beyond the funding cycle and do an SEIE?
  • What would help to make this type of evaluation more widely undertaken?
  • If you have done a post-project evaluation, how did you do it? What were some of the barriers you faced and resources you were able to draw on to overcome them?

 

How can we lead together to Value the Voices of those we serve!?

(Reposted from https://medium.com/@WhatWeValue/leading-in-challenging-times-sustained-and-emerging-impacts-evaluation-seies-617b33bf4d27#.ec7fcg4ty)

 

 

Sources:

[1] Foulkes, I. (2017, February 27). Is there a US diplomacy vacuum at the UN in Geneva? Retrieved from https://www.bbc.com/news/world-europe-39080204

[2] Cekan, J. (2015, March 13). When Funders Move On. Retrieved from https://ssir.org/articles/entry/when_funders_move_on

[3] UN DESA. (2011, March 2). Lasting impact of sustainable development. Retrieved from https://www.un.org/en/development/desa/news/sustainable/sustainable-development.html

[4] UN Statistics Division. SDG Indicators: Global indicator framework for the Sustainable Development Goals and targets of the 2030 Agenda for Sustainable Development. Retrieved March, 2017, from https://unstats.un.org/sdgs/indicators/indicators-list/

[5] Cekan, J., Zivetz, L., & P, R. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE

 

Making up your mind. Prioritizing and making it happen

Making up your mind. Prioritizing and making it happen

 

* As Dr. Martin Luther King Jr. said, "every man must decide whether he will walk in the light of creative altruism or in the darkness of destructive selfishness".

 

 

 

 

 

 

 

 

Source: https://shanemcdonnell45.wordpress.com/tag/darkness-into-light/

* Our President, Barack Obama said in his farewell speech, "change only happens when ordinary people get involved and they get engaged, and they come together to demand it".

* OXFAM International demanded changing shocking inequity: "just eight billionaires own the same wealth as the 3.6 billion people who make up the poorest half of humanity".

* Caroline Heider of the World Bank's IEG asked we examine how we evaluate long-term impacts: "current considerations of efficiency, cost savings, or cost-benefit analyses are challenged to take long-term impacts into account".

 

What do you want to prioritize and demand of international development? In these times of shifting priorities in powerful nations, where politicians are questioning the needs of those whom many of us have been serving, what do you want to demand? What issue do you prioritize, and want to move forward?

 

I choose to prioritize sustained impact driven by country-nationals. Why? I grew up in large cities, and when I first worked in Africa’s Sahel desert 25 years ago, the herders and farmers making a living from the arid pastures and sandy soil, with wells 100 feet deep astonished me.

 

Without them, I’d last 3 days out there. They were the experts.

 

I always assumed we measured ‘sustainable’ development in work with such herders and farmers, but in 2013 I founded Valuing Voices after I began to see how rarely we return to evaluate what remained after our foreign aid projects stopped.

 

Reviewing  thousands of “ex-post” or “post-project” documents in 30 organizations’ public databases, Valuing Voices has found the vast majority of documents only suggested a post-project be done, a small proportion were desk studies and fewer than 1% were original fieldwork post-project evaluations of sustainability. In these 370 post-project (ex-post) evaluations, development workers asked partners and participants what was still standing, showed what succeeded or failed and what unexpected successes participants created themselves from what we left behind.

 

Returning to learn, consulting our participant-experts seems so common sense as they are the ones that can tell us what we should replicate, adapt or abandon.  In 2015 research we found only three World Bank IEG evaluations that asked participants their views in a methodologically clear way (out of 33 post-project PPAR evaluations), and only one was perceived as successful.  On the other hand, in 2014, IRIN highlighted Rwanda’s very successful community based nutrition solutions, replete with participant voices.  We have found 23 ‘catalytic’ (mostly NGO) organizations having done one or more (ex-) post-project evaluation that include participant input and each of them is filled with excellent lessons for doing ‘development’ well now and after closeout. Yet what are any of these organizations doing differently and why are so few doing more? Why do donors seem to care so little about sustained impact that such studies are so rarely funded by them, and NGOs use private funds? That is what drives me.

 

A seminal book, Time to Listen asked 6,000 such participant-experts in 20 countries what they wanted foreign aid to look like. “Very few people call for more aid; virtually everyone says they want “smarter” aid…. A majority criticize the “waste” of money and other resources through programs they perceive as misguided or through the failure of aid providers to be sufficiently engaged… [it is] a supply-driven approach that squeezes out the views of the recipients, and a focus on spending – both volume and speed, which undermines aid’s ability to listen, learn and adapt to local contexts.”

Sobering stuff.

 

While Valuing Voices is not profitable (yet?) and growth is slow, I continue to evaluate and advocate, believing that designing, implementing, monitoring and evaluating for sustained impact by our true clients is key to successful work life well spent.

 

We need a sustained impact mindset.

 

We are getting there. Better Evaluation just featured our Sustained and Emerging Impact Evaluation (SEIE) approach as a new theme in evaluation. OXFAM and Save the Children recently wrote “The Power of Ownership: Transforming US Foreign Assistance” (2016). They ask: “country ownership is at the core of effective development… as the United States transitions to a new President and new leadership for development cooperation, how will the next administration build on current successes and chart a path forward?“  I fear the answer, as it takes trust and interest in countries’ capacity to chart their own way forward.  USAID (and maybe other donors?) are ready to help. USAID alone has some done some exciting work recently through USAID Forward’s local partners (e.g. Afghanistan has done this in depth) and it has looked at Local Systems. Food For Peace’s strategy includes sustainability.

 

What will be a priority in 2017 onward? What each of us creates will remain.

 

The powerful Sidekick Manifesto beautifully proposes this new core belief which we can each espouse, that “Local leaders with local solutions to local problems” will end poverty. We will not.” We can, however, “always be listening, learning and seeking a deeper understanding…” I am delighted to be a sidekick in projects that prioritize participant and partner views, for that is how they end poverty.

 

What do you want to prioritize and create? What is so vital for you that you must work on it?  What has been neglected?  What difference do you want to make?  GO!