by Jindra Cekan | May 18, 2015 | Dissemination, Feedback loops, innovation, local capacity building, Local Participants, post-project evaluation, Sustainability, Sustainable development
Longing to do an Ex-post Sustainability Evaluation? How to support this work…
Just back from Niger where Catholic Relief Services, Rutere Kagendo and I are doing a fascinating post-project evaluation. Fresh on my mind is the commitment we all had to make to get this quite ground-breaking research going. Here is the full report, but there are three kinds of conditions we found were integral for success: client-ValuingVoices match; project and site selection; and resources.
Client – Valuing Voices match:
-
The study needs to be appreciated as innovative, adding to the program quality and learning of the organization so funding is provided and there is in-house interest in the findings;
-
The local office needs to allocate staff and technical time to support the study technically and logistically (see below);
-
Shared clarity is needed among all involved that such a study looks for self-sustained activities and outcomes. While there are lessons that emerge about the quality of implementation, its focus is what participants and their country-partners could continue themselves after project close-out and withdrawal of resources. It also can include lessons about what the local non-profit and the national stakeholders are doing to support community success (or not) and unexpected outcomes. Our clients need an openness to honestly seeing what was not sustained and exploring why;
-
While Valuing Voices provides expertise given review of the handful of post-project and exit evaluations that exist, client is interested in sharing findings and advocating to donors to fund more of these studies;
-
Disseminating findings internally where a possibility of learning from this evaluation to support similar current implementation; research could help similar project learning, lessons for country nationals such as Ministries;
-
Prioritizing local capacity – Valuing Voices believes in using regional M&E capacity to do the work; where possible we partner with regional evaluators while also building capacity within our client's staff to carry out such work;
-
Sharing and discussing findings locally: Valuing Voices believes knowledge learned needs to be shared in immediate feedback loops. We present: a) to each village after each site’s research; b) to local key partners and representatives from each village at the end of the qualitative Rapid Rural Appraisal findings; c) to the non-profit in-country at the end of qualitative research and d) internationally to headquarters at the end of the combined analysis of the qualitative and quantitative research with findings in the final report.
Project and Site Selection:
-
Non-profit programming projects has been closed out for at least two years and no more than seven years (for recall);
-
No other NGO has done very similar work in the region in the intervening years;
-
The region selected is representative of the project as a whole (e.g. agro-ecological zones, economic/ livelihood/ health, educational or other sectoral criteria);
-
Research areas are secure and safe (e.g. from civil unrest, severe drought/ floods, epidemics, to the degree possible);
-
Timing does not interfere with urgent priorities of those involved with the study (e.g. livelihoods are not jeapordized in communities, holidays are kept, other technical work is not disrupted).
Resources (Time, Material and Project Expertise):
-
Time: the research is qualitative followed by quantitative, coming to 80-90 days of qualitative and quantitative research overall (roughly 5 weeks of fieldwork in teams of 4-10, analysis, report-writing and presentation;
-
Project and evaluation documents are available to inform and contextualize approach including activities, outcomes and projected impacts;
-
Data: key to the fieldwork are village and participant lists from pre-closeout days so participants can be interviewed both during a Rapid Rural Appraisal and a follow-on household survey;
-
Internal/external sectoral staff and at least one past project staff are part of the team to inform and ‘ground truth’ research;
-
Logistics support is provided by the client; from vehicle/driver and lodging support in the field to materials such as mobile phones, flipcharts, photocopying and advances;
-
A consultant or staff prepares the sites before the research teams come, e.g. to confirm communities are willing to be visited (each visit will be 2-5 days) and to identify participants and partners still there;
-
Partners familiar with the closed project can be identified so they can be interviewed by the research team;
-
Local language expertise is needed, e.g. translator to local language, as well as data entry personnel afterwards.
-
While Valuing Voices provides the technical lead experts and statistical back office analysis, including sampling and rigorous analysis, senior non-profit staff are needed in-country for contextualization and input on preliminary findings, as well as senior technical staff to review the final product;
-
A home for findings and on the road for dissemination: good knowledge management is needed for data retention, for the findings to have a sustainable ‘home’– be that info-graphics and print copies distributed to villages and partners or online repositories created that are language-accessible both nationally and by foreign donors; webinars, conference presentations etc are needed to optimize the learning via sustainability dissemination campaigns.
Much of this is needed for the research to be the best quality and yield the highest results. It is exciting learning is to be had from not only what communities and supporters could sustain but what they exceeded or dropped! Consider doing one to see how post-project sustainability research can improve your current implementation, future design, and long-term self-sustainability!
by Jindra Cekan | May 1, 2015 | Job Training, Umzi Wethu, Youth
Embodying the human approach: Umzi Wethu (Part 1)
Guest Post by Paul Longe of Umzi Wethu
In “Ideas we’re not ready for are the ones we need most,” posted here last December, Barbara Geary Truan highlighted the work of the South African Wilderness Foundation’s Umzi Wethu programme as “embodying the human approach” to development. As a core member of the Project Management Team at Umzi, I was struck by these words as they so aptly describe the heart of our approach. We’re often at a loss to describe ourselves as we find our focus locked on the daily realities of the “who” rather than the “what” and “how”.
Umzi Wethu fulfils the employability potential of resilient, motivated youth displaced by HIV/AIDS and poverty by using the power of the wilderness, promoting personal wellness in a nurturing home context, providing credible training, and securing sustainable job placements in hospitality and eco-tourism establishments while extending the program's social outreach to others.
I’m often asked how we go about ensuring the ongoing success of the programme. At the time of writing, Umzi Wethu continues to boast an 85% job placement rate for over 200 graduates and is currently on its 17th intakes of students. There is no short answer about how we do this, but in subsequent posts I will reflect on some key aspects of the work which go into the long term sustainability of what we do.
When Umzi Wethu began in 2007, it was ground-breaking for a number of reasons, not least of all in its intention to produce seriously life changing outcomes with a fairly small number of specially selected youth in a relatively short space of time. This highly concentrated approach is in contrast to what is usually expected in development work, where reaching as many people as possible through service delivery is often where the work is focused. If we think of this in terms of Maslow’s hierarchy of needs, the classic development approach seeks to ensure the basic physiological and survival needs of food, water, shelter and primary health care which is much needed but often leaves individuals dependent on others to provide for themselves.
Umzi Wethu began as a bold leap to leverage opportunities for employment in the high status world of luxury game reserves and hotels for the benefit of disenfranchised young people who would not ordinarily be encouraged to aspire to these heights. Perhaps there is something unique in the organisation’s belief in young people’s ability to reach beyond their wildest dreams after only one year. We’ve subsequently heard stories from our students who shared how the experience of training and working in prestigious establishments provided a huge boost to their own personal sense of self-esteem and achievement which had previously been eroded by a perceived lack of access to even the most basic opportunities. One thing that we hold to is an unashamed belief in just how amazingly far young people can go in 12 months when given a caring, supportive home environment and opportunities to get hands on experience in the workplace.
With Umzi, the Wilderness Foundation creatively designed a fully holistic model – residential, vocational, emotional, environmental, etc. – which aimed to tackle almost every aspect of each individual student’s life over the course of one year. In doing so, the Foundation took on the challenge of touching on all levels of Maslow’s model in order to provide the foundation for a level of self-actualisation not usually associated with young people from such vulnerable backgrounds. You see what I just did there: they are not “vulnerable youth”. They are incredibly motivated and resilient youth who have been stuck in stories of vulnerability and despair – a conversation we have with our students from day one.
Such a perceptual turnaround is no mean feat, and while a well conceptualised project model together with carefully constructed monitoring, evaluation and learning processes was key, there is definitely something more to the Foundation’s approach than meets the eye. How we view the “who” aspect of our work has been a key factor in this success from the outset. It touches on the very intentional way in which we come together as partners to create stories of belonging and self-esteem that lead our students to a future of success and achievement which they’d not previously believed possible.
In the next piece I will unpack the “who” in more detail by focusing on a few key observations which have been made about the personal aspects of the work we do.
Paul Longe is a qualified Counselling Psychologist working in Port Elizabeth, South Africa. He has been working with the Wilderness Foundation, a global conservation organisation, on the development and implementation of innovative and holistic socio-economic development projects for vulnerable youth in South Africa since 2008.
by Jindra Cekan | Mar 23, 2015 | Accountability, Aid effectiveness, Catholic Relief Services (CRS), Center for Effective Philanthropy, EU, ex-post evaluation, Hewlett Foundation, IATI, JICA, Keystone, OECD, post-project evaluation, SSIR, Sustainable development, USAID
When Funders Move On: Donors and nonprofits need to learn more about how to help program participants keep progressing after the support ends.
Imagine standing in Detroit or South-Central Los Angeles. A team of experts has come to help you out of grinding poverty. Some of these experts specialize in credit issues, others in education or health or gardening. They have funding for three years, so they set up offices, create participant lists and prioritize problems to tackle. They give you seeds and loans, and advice. And you—and others from your neighborhood—begin creating small businesses and home gardens. You and other adults learn about infant nutrition; children who live in your area get free school materials; teachers at your local schools receive extra training. Everyone begins to do better.
A year and a half passes. Another expert arrives to find out how things are going relative to the team’s projections. Some businesses are succeeding, others have faltered; some gardens are flourishing, others are neglected. You participate in a focus group, and you answer questions optimistically.
At the three-year mark, many of your neighbors are participating in this project, and tangible successes appear to be spreading. But suddenly, the experts are packing their boxes. The project office closes. The initiative has supposedly been “handed over” to the community. No one who worked for the project comes back.
No one comes back. This is the state of affairs for too many so-called “sustainable international development” initiatives around the world, and it has to change.
As Gugelev and Stern brilliantly note in a recent SSIR article, we must be transparent about our “endgame”: Too many international development projects are bound to fixed endings and “fail to reckon with the gap between what the nonprofit can achieve and what the problem actually requires.” Due to fixed funding requirements, donors often leave when the calendar tells them to, whether or not a project has achieved the desired impact. And according to Valuing Voices research, they don’t even go back to assess the outcomes of their work or consider what (if anything) might help progress continue!
Since 2000, the US government has spent more than $280 billion on bilateral and multilateral assistance; the EU has spent $1.4 trillion. Just in 2002, US foundations, businesses, and NGOs spent more than $34 billion overseas. And while most taxpayers believe that this spending supports “sustainable development,” our research shows that 99 percent of the nonprofit grant and for-profit contracted projects these funds enabled were not evaluated after the funding concluded. Unfortunately, this continues: In 2014, the US spent $20 billion and the EU spent $80 billion on program assistance without any plan for post-project evaluation. This does not mean the projects are not sustainable; we simply do not know.
In fact, the United States Agency for International Development (USAID)—once considered the leader in post-project evaluations assessing relevance, effectiveness, efficiency, and sustainability—has managed only one post-project evaluation in 30 years (due later this year). And although thousands of documents appear in multilateral donor database searches as evaluations, most are “desk studies”—conducted remotely and not based on new fieldwork. Of these, only a few include feedback from program participants (leading the way are Japan’s International Cooperation Agency and the UN’s Organization for Economic Co-operation and Development, which have systematically done post-project evaluations).

There is usually terrific monitoring and evaluation during project implementation (red) and evaluation at start-mid-end (green), virtually no one returns afterwards (blue).
Why don’t we do a better job of following up? Are we afraid of the possibility of seeing poor results? If so, it’s time to face that fear. In some cases, we will surely see good results or even unanticipated positive impact—we’re missing that too.
Hewlett Foundation’s Fay Twersky implored nonprofits to “systematically solicit feedback from intended beneficiaries” in an SSIR podcast on Monitoring and Evaluation (M&E). We agree. We need to know more than whether those participants’ situations are improving while a program is in full swing, and we need to know what it will take for things to continue to improve after the funding goes away. Imagine the cost efficiencies we would gain by replicating activities that they could sustain. Imagine the cost-efficiencies and productivity if we prioritized activities with the largest sustainable return on investment (ROI). Now that is something impact investors could buy into.
There are positive signs on the horizon. According to Keystone Accountability, an increasing number of nonprofit organizations are committing to “making governments, NGOs, and donors more responsive to the needs of their constituents.” And funding that supports the idea of using participant feedback to improve programs and make them sustainable is on the rise, as Center for Effective Philanthropy’s “Hearing from Those We Seek to Help” and the Fund for Shared Insight have noted. All of this could yield a hugely different array of endgames that are sustainable by communities that can perhaps later even select development aid offers based on past effectiveness.
But there is still much to do. We offer the following recommendations to project implementers and donors, based on our own experience, and on our observations of several initiatives where we have seen project participants independently continuing and adapting work that was begun with external support:
- Shift the development model from what donors and implementers think would be best to what the intended participants think is best. Design projects with them, and mandate that request for proposals (RFP) design involves communities.
- Document, share, and discuss what was most sustained and what participating communities and local partners can do to sustain projects, and how we can sustainably support them through design and implementation to make them more effective.
- Require a plan for transitioning to sustainability after projects close for any project that uses more than $1 million dollars. This should include handover plans to local nonprofits, with training and financial support; training for communities on how to manage the sustainable activities it prioritizes; financing mechanisms for those activities; and report sharing in IATI open-data format, with project data saved and stored in the cloud for global access.
- Do post-project sustainability evaluations on all these projects, and discuss the results widely with other funders, the government, and the private sector, including how to feed back lessons into future design.
- Advocate for participatory input in all evaluations. This input should make up 30 percent of future evaluation findings (now far less).
- Consistently solicit feedback from local communities through national evaluators, both during and after projects, to better understand how the program you’re running or supporting from afar is working on the ground. Invest in building the national capacity and systems needed to make that feedback helpful for all stakeholders, including national governments.
- Advocate for extensive civil society input into the United Nations’ Sustainable Development Goals so they serve our participants’ visions for the world they want.
As Peter Kimeu of Catholic Relief Services said to me, “It will be sustainable development if the people at community level are involved in designing and delivering their own dreams of development.”
**************************************
Jindra Cekan (@WhatWeValue) is founder of Valuing Voices, with 28 years in international development design, monitoring, and evaluation. She has a doctorate from the Fletcher School of Law and Diplomacy; was a University of Cambridge Fellow; and works with foundation, nonprofit, and for-profit clients.
by Jindra Cekan | Mar 14, 2015 | Accountability, Aid effectiveness, Evaluation, Food for Peace (FFP), Food security, IATI, International aid, Local Participants, Maternal Child Health, MCHN, Open Data, Project design, Sustainability, Sustainable development
Are We Done Yet?
When are we off the hook, so to speak, for the well-being of the participants whom we said we'd make healthier, better fed, more educated, safer, etc?
America’s Agency for International Development (USAID) is the main channel for international development aid. It is also an organization interested in learning from its programming and numerous contracts support such work. One such contract by FHI360/FANTA was Food for Peace tasking them to review the agency’s Title II development food aid from 2003-2009 covering 28 countries. This Second Food Aid and Food Security Assessment (FAFSA-2) Summary found that such programs can “reduce undernutrition in young children, improve health and nutrition outcomes, and increase access to income and food” and also found practices that did not work well.
While USAID has made enormous strides in the intervening six years on monitoring and evaluation (I was a consultant to USAID/PPL/LER from 2013-14), excellent recommendations that would support great, sustainable programs are unfulfilled:
Recommendations #1, 4 “USAID/FFP should develop an applied research agenda and sponsor studies that focus on the implementation of Title II programs in the field to better define what works and what does not…. [and] should select the review panel for new Title II applications… and give reviewers a ‘cheat sheet’ on interventions and approaches that USAID/FFP is and is not interested in funding because they work better or do not work as well, [and] provide this same information in the Request for Assistance” [Request for proposals].
Yes, all across our industry there is little learning from past evaluations for future design and Valuing Voices believes local participants and stakeholders need to be consulted to tell us what (still) works and what they want more of not only during implementation but long after. Their voices must support great design, as it’s their lives we go there to improve; they must be involved in the design of these original requests that non-profits design and fulfill. Further, the study found that only 1/3 of all evaluations were included in USAID’s database[1], and as Valuing Voices’ partner Sonjara has written in our blog, aid transparency requires data retention and access for learning to happen.
Recommendation #3 “USAID/FFP should include options for extensions of awards or separate follow-on awards to enable USAID/FFP to continue to support high-performing programs beyond five years and up to ten years… [as] longer implementation periods are associated with greater impact.”
This would address the ‘how much impact can we accomplish in 1, 3, 5 years” question that many of us in international non-profits ask ourselves. Finally, the graphic below is self-explanatory – USAID sees its role ending at close-out.
The crux lies in their honest statement: "It was beyond the scope and resources of the FAFSA-2 to explore in any depth the sustainability of Title II development programs after they ended." While they state that there is merit in having impact while you intervene, such as "having a positive impact on the nutritional status of the first cohort of children is of immense benefit in its own right", they go on to say that "ideally, one would like to see mothers continuing positive child feeding practices and workers continuing to deliver services long after programs end… [yet] whether the [maternal child health and nutrition] interventions are sustainable beyond one generation is unknown and would require research." This is because funding is pre-programmed, fixed to end within set 1, 3, 5 year increments, and no one goes back to learn how it all turned out. This is what most needs to change, this illusion that what happens after closeout is no longer our issue, that the ‘positive impact’ we had while there is enough.
They are not alone. I think of NORAD, the Government of Norway's development arm as very progressive. So I went to NORAD's website and searched for 'ex-post' (we do a lot of that at ValuingVoices). So like our World Bank blog on finding real ex-post evaluations, many many things are considered 'ex-post', including one actual evaluation in Palestine with fieldwork which asked participants and a few that looked at institutional sustainability. Many of the 100+ 'finds' were actually documents recommending ex-post. Typical of our other searches of other donors. I emailed NORAD whether there were more with participant voices, yet they assured me they did them. Maybe our problem is in definitions and taxonomy again. Maybe we should call them post-project participant feedback?
Most of my colleagues would agree that the sustainability of activities aimed at making communities food secure in the long-term and independent of aid is a shared goal, one which short-term assistance aimed at huge impacts such as to ‘make communities food secure’ and ‘sustainably decrease malnutrition’ (common proposal goals) is unrealistic. We need participant voices to teach us how well we served them. We need to return, learn “what works and what does not”, and Value Voices in true sustained partnership. We all look forward to being done.
[1] “Another major obstacle to transparency and learning from the Title II program experience was the fact that only one-third of the final evaluations were publicly available on the Development Experience Clearinghouse (DEC), despite the requirement that Awardees post them to the DEC…. [There was a lack of] cross-cutting studies or in-depth analyses of Title II evaluation results to advance organizational learning [and] much greater use could be made of the evaluation data for systematic reviews, meta-analyses, secondary analyses, and learning.”
by Jindra Cekan | Mar 6, 2015 | Accountability, Evaluation, Evidence-based policy, InterAction, International aid, Local Participants, Participants, Project design, USAID
A Missing Piece In Local Ownership: Evaluation
(Reblog from http://www.interaction.org/blog/missing-piece-local-ownership-evaluation Grino and Levine)
Ten years ago, ownership was established as a key principle of aid effectiveness. Although understanding of ownership has evolved since then – most significantly, as something that involves not just governments but all parts of society – today the focus is not on whether ownership is important but on how we can move ownership from principle to practice. To date, these conversations have primarily concerned how to make ownership a reality in program design and implementation. InterAction supports these efforts, but believes they need to go one step further. As we argue in our new briefing paper, the local ownership agenda must extend to all parts of the program cycle – from design all the way through evaluation.
Including those meant to benefit from international assistance (we use the term “participants”) in deciding what should be done and how it should be done is critically important for effectiveness and sustainability. Organizations, and some governments, also increasingly recognize the value of hearing directly from participants and citizens about how well something is being done. This can be seen in the growing use of feedback mechanisms and the establishment of initiatives promoting social accountability. Including participants in evaluation decision making is just as important. Particularly when participants have lacked ownership at other stages of an intervention, evaluation serves as a last opportunity for them to weigh in.
Despite the widespread acceptance of the principle of local ownership, evaluations continue to predominantly respond to the demands of donors, focusing on how funds are spent and the degree to which the results donors or implementers value are achieved. By only taking into consideration the values and interests of some stakeholders (primarily donors and external actors) in evaluations, organizations are missing a critical perspective on an intervention’s results: the views of the very people the intervention was intended to assist.
When participants are involved in evaluation, more often than not they serve as data sources, and perhaps as data collectors. Very rarely do we find examples of participants involved in deciding the questions an evaluation will ask, determining the criteria that will be used for judging an intervention’s success, interpreting results, or shaping recommendations based on evaluation findings.
A concern frequently raised about including participants in evaluation decision making is that their clear stakes in evaluation outcomes and potentially their lack of evaluation capacity could lead to biased and unreliable results. Yet it is important to acknowledge that everyone involved in an evaluation has values, interests, and capacities that affect how they approach an evaluation. Including participants’ voices adds a greater diversity of perspectives to an evaluation and the interpretation of findings, thus reducing bias.
We recognize that the road to local ownership in evaluation is just that: a road, not something that can be achieved instantly or that is possible in all cases. For that reason, we recommend that organizations take an incremental approach to pursuing local ownership in evaluation, focusing on the critical steps that can be taken along the way to increase the role of participants in evaluation processes.
As organizations seek to increase participants’ ownership in evaluation, they must consider:
-
Who to include as co-owners in an evaluation;
-
In which aspects of an evaluation participants need to be involved (we provide a list of possible evaluation activities related to designing the evaluation, collecting and analyzing data, determining findings and recommendations, and disseminating and using evaluation results); and
-
The nature of participants’ involvement (with the goal of moving from informing or consulting participants to including participants as partners in evaluation decision making).
Getting to local ownership in evaluation requires making progress on all three fronts.
Ultimately, all actors along the aid chain – from donors to international NGOs to local partners – must believe in the value of including participants as co-owners in evaluation. Once in place, this commitment must be complemented by investing in staff’s capacity to effectively involve participants in evaluation decision making, and in strengthening participants’ own capacity to engage. As in any other participatory process, participants must also trust that their input will indeed influence policies and practice. Including participants in this way is another way to signal that we truly view them as partners, rather than beneficiaries.
By Laia Grino, Senior Manager for Transparency Accountability and Results, and Carlisle Levine, Ph.D., Senior Advisor, Evaluation (Consultant)