by Jindra Cekan | Jul 7, 2020 | Accountability, ex-post evaluation, International aid, Participation, post-project evaluation, Sustainable development
On June 24th under GLocal’s UNConference, “Co-creating our future stories of hope and action”, Jindra Cekan, Holta Trandafili, and Isabella Jean presented their work on sustainability evaluations and exit strategies via local voices. We chaired a 2-hour discussion session on the following topics:
- Sustainability of global development projects and exit from them,
- The importance of valuing local partners’ and participants’ voices,
- How to embed ex-post evaluation of sustainability into the project cycle,
- How expectations, benchmarking, and early joint planning in exit strategies, as well as considering long-term ownership & relevance will support projects to be sustained locally, including questioning who will maintain results,
- Considering power dynamics between donors and ultimate ‘beneficiaries’ and
the value of the impact of the project from a variety of perspectives, and more
Here is the recording of our presentation or see just the PowerPoint presentation. We harvested lessons from our three presentations:
Jindra Cekan:
· Fear of learning about failure in our global development industry – INGOs are “waiting for a successful enough project” to commission ex-post project sustainability evaluation
· It needs to be a culture of learning, not a culture of success.
· Lack of transparency in sharing program evaluation results with communities and local government is widespread, and even more rarely do we come back after many years and share learnings
· Participatory approaches are vital, listening to participants about sustained impacts is key
· It is never attribution, always contribution. To isolate impacts, we need to look for project sites that haven’t had multiple other organizations overlapping through all phases.
· Building sustainability planning throughout the project cycle is key – but often doesn’t happen
Holta Trandafili:
· Sustainability needs to be planned to be researched, including evaluating why or why not were project elements sustained, and why? What has the project done to enable communities to sustain improvements?
· Expectations of sustainability need to be more modest (as most results are mixed good/bad)
· We need to ask: How are you defining and measuring sustainability – for how long should the results last? Among how many participants? Have you set benchmarks for success?
· We should expand your toolbox on methodology to investigate sustainability. Stories of success are one of a myriad of methods used, including mixed-methods, cost-benefit, etc.
· Start with the need for learning not [just] accountability
Isabella Jean:
· Sustainability investigation/evaluation/learning should be mindful that this is NOT about projects. It is about people.
· We have a system that focuses on gaps and needs to be filled vs. existing capacities’ structures to be reinforced. How can our work on measuring sustainability bring this to light and call it out, so that we change the norm?
· Planning for sustainability requires the insight to integrate resources and experiences of outsiders with the assets and capacities of insiders to develop context-appropriate strategies for change
If you would like to discuss this with any of us, please send us comments and we’re happy to respond. Thanks again to the G-Local UnConference team!
Below please find our bios:
Jindra Cekan/ova has worked in global development for 33 years focused on participatory design and M&E for global non-profits. She founded Valuing Voices 7 years ago. For details, see: Valuing Voices Founder
Holta Trandafili is the Research, Learning, and Analytics Manager with World Vision US and has been leading field research, monitoring, and evaluation since 2007. She has led sustainability measurement studies for World Vision programs in Uganda, Kenya, Sri Lanka, India, Burma, and Bolivia. Her areas of expertise and interest include program and community groups’ sustainability measurements; agency-level measurements; empowerment approaches to development; integrated programming; local capacities for peace; gender analysis; and outcome monitoring. Currently, Mrs. Trandafili serves as an Advisory Committee member for InterAction’s Effectiveness and Program Evaluation Working Group and chairs one of the sub-working groups under The Movement for Community-led Development.
Isabella Jean supports international and local organizations and funders to document promising practices, facilitate learning and strengthen capacities for conflict sensitivity, peacebuilding and humanitarian effectiveness. She has facilitated action research, collaborative learning and advisory engagements in over 25 countries, and serves as an advisor to policymakers, senior leadership and program teams. Isabella co-authored the book, Time to Listen: Hearing People on the Receiving End of Aid and developed practical guidance to support accountability to communities, listening and feedback loops, and responsible INGO exits. She teaches graduate-level courses on aid effectiveness, program strategies and M&E of peacebuilding at Brandeis University’s Heller School for Social Policy and Management. Previously, Isabella directed training at a community organizing network and conducted policy research for the Institute for Responsive Education, UNDP, and Coexistence International.
by Jindra Cekan | Nov 10, 2019 | Aid effectiveness, civil society, ex-post evaluation, Ghana, Impact, impact evaluation, Indonesia, Maternal Child Health, MCHN, NGOs, post-project evaluation, sanitation & hygiene, Senegal, Sustainable development, USAID, water/ sanitation
Learning from a river of ex-post project evaluations and tools… Thanks USAID!
Dear ex-post aficionados. It’s raining ex-post project evaluations. Here’s hoping learning from such evaluations in water/ sanitation, maternal/child health and even capacity building/ peacekeeping, and their number increases!
1. WATER/ SANITATION & HYGIENE:
USAID has a series of six ex-post evaluations of the water/ sanitation and hygiene sectors since 2017! What is exciting is that they are also looking to the future. These evaluations will “provide insight into what happens after an activity ends, and how to mitigate challenges in future programming, potentially. The series will inform USAID’s WASH activity design and implementation and contribute to a larger sector discussion on achieving sustainability.”
The E3 water division (Water CKM ) took sustainability on as their strategy and have made great strides these last two years. They have done five ex-post project evaluations, cited below, and MSI has completed one more wat/san/ hygiene ex-post evaluations, specifically:
Madagascar Rural Access to New Opportunities for Health and Prosperity (RANO-HP) – Published June 2017
The first evaluation in the series explores the sustainability of the sanitation and hygiene components of the RANO-HP activity, implemented in 26 communes from 2009–2013.
Indonesia Environmental Services Program (ESP) – Published August 2017
The second evaluation in the series examines the sustainability of water utility capacity building, microcredit, and financial outcomes associated with the ESP activity, which was implemented from 2004–2010.
Ethiopia Millennium Water Alliance (MWA-EP) – Published May 2018
The third evaluation in the series examines the long-term sustainability of outcomes related to rural water point construction, rehabilitation, and management, as well as participatory sanitation and hygiene education and construction related to the MWA-EP activity, implemented in 24 rural districts between 2004–2009.
Financial Institutions Reform and Expansion–Debt and Infrastructure (FIRE-D) – Published September 2018
This evaluation is the fourth in the series. It examines how urban water and sanitation services in India have changed since FIRE-D closed and to what extent policies, practices, and financing mechanisms introduced through FIRE-D have been sustained.
Millennium Water and Sanitation Program in Senegal (PEPAM/USAID) – Published July 2019
The fifth ex-post evaluation in the series looks at the PEPAM project (Programme d’Eau Potable et d’Assainissement du Millénaire au Sénégal), implemented from 2009–2014 to improve sustainable access to WASH in four regions of Senegal.
USAID-funded by MSI: USAID/Ghana’s Water Access, Sanitation, and Hygiene for Urban Poor (WASH-UP)– published Nov 2018
Also USAID and Rotary International developed a WASH Sustainability Index Tool, “to assess a WASH activity’s likelihood to be sustainable according to the following factors: availability of finance for sanitation; local capacity for construction and maintenance of latrines; the influence of social norms; and governance.” This is similar to what we learned from USAID/ FFP/ Tufts/ FHI360 12 ex-posts that resources, capacities, motivation and linkages (aka partnerships, including governance) are vital to sustaining outcomes and impacts.
It will be interesting to see whether they examine the other ex-posts for excellent lessons, as they have the Senegalese evaluation:
- “Whether or not to subsidize sanitation access …Based on this evaluation’s findings and exploration of the literature, subsidies can help improve the quality of household latrines, but increasing use of those latrines remains a challenge.
- In contrast, CLTS (a nonsubsidized approach) is often credited with increasing use of unimproved latrines, but serious questions linger about quality and long-term sustainability of the latrines built after CLTS triggering, particularly as it relates to moving up the sanitation ladder. This evaluation… provides the opportunity to examine the potential value of a hybrid approach….
- The handwashing results suggest that low-cost, low-quality handwashing stations such as tippy taps do not lead to sustained behavior change. It may be worth considering hygiene investments that reduce the behavior change burden on targeted beneficiaries.
2. MATERNAL/ CHILD HEALTH & NUTRITION:
“Sustainability of a Community-Based CHOICE Program to Improve the Health and Nutrition Status of Mothers and Infants in Indonesia,” The report focused on whether the USAID-funded CHOICE program had left sustainable impacts: improving the health and nutrition status of children under the age of five, as well as the health status of pregnant and lactating women and mothers or caretakers of young children in the Pandeglang District of Indonesia. “After examining the data collected from the PSS, the researchers found that there were significant improvements in many indicators—such as births attended by skilled personnel, the treatment of diarrhea, and the nutritional quality of food fed to infants—in the six years after the CHOICE program ended. However, despite these improvements, the researchers found no significant statistical differences between villages that received the CHOICE program interventions and comparison villages, which did not. This speaks to using such a comparison methodology to focus on actual contribution and rule out the “rising tide lifts all boats” phenomenon.
3. CAPACITY DEVELOPMENT & PEACEBUILDING:
USAID’s Regional Office in Thailand evaluated its capacity building and peacebuilding program 1.5 years ex-post. While civil society was strengthened and there were inroads made on peacebuilding, “many interventions initiated during Sapan did not continue post-Sapan, although some did remain. For example, “stakeholders cite evidence of continuing to use some governance tools in local governance related to service delivery [although] because of limited financial resources after Sapan ended, they had to change some of their interventions and reduce the range of people they could include. There are lessons for whose capacities are built, two-way feedback loops with local partners, using local organizations such as universities to sustain training, planning sufficient time for partners to internalize training lessons, etc.
4. USAID FUNDED GUIDANCE:
‘Impact Evaluations’ have a new focus on long-term impact, rather than effectiveness during implementation (which was at least the original intent of impact evaluation in the 1980s)! In September 2018, USAID and Notre Dame issued a Guide for Planning Long-term Impact Evaluations as part of the Utilizing the Expertise of the ERIE Program Consortium. The guide covers the difference between traditional impact evaluation designs and data collection methods and how to apply them to long-term impact evaluations (LTIE). It also shares examples across a range of sectors, including later evaluating past impact evaluations, which ended before final evaluation.
Finally, in new 2018 USAID guidance, ex-post evaluation is clarified as the source of the sustainability of services and benefits. USAID clarifies that “questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation, a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be, and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project’s services and benefits were sustained.”
Such richness that we can learn from. Keep the momentum going on the 99% of all global projects yet unevaluated ex-post, and change how we fund, design, implement, monitor and evaluate global development projects!
by Jindra Cekan | Oct 19, 2018 | Doing Development Differently, Ethiopian Red Cross, European Evaluation Society (EES), ex-post evaluation, Exit strategies, foreign aid, Impact, impact evaluation, International aid, international development, NGOs, OECD, OXFAM, Participants, post-project evaluation, SDGs, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Sustained and Emerging Impacts Evaluations (SEIE)
Assuming Sustainability and Impact is Dangerous to Development
(+ OECD/ DAC evaluation criteria)
We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impact such as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.
INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions” [1]. They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact” [1]. Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer. Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable” [2].
How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.
Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).*
There are remarkable assumptions routinely made. Many projects put sustainability into the proposal, yet most close out projects in the last 6 months. Rarely do projects take the time to properly phase down or phase over (unlike CRS Niger); many exit ceremonially ‘handing over’ projects to country-nationals, disposing project assets, and leaving only a final report behind. Alternatively, this USAID Uganda CDCS Country Transition Plan which looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit [3]. Maybe they will measure progress towards that goal and orient programs toward handover, as in the new USAID “Journey to Self-Reliance” – we hope! Truly, we can plan to exit, but only when data bears out our sustained impact, not when the money or political will runs out.
As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters” [4]. In fact, most ‘impact evaluations’ actually test effectiveness rather than long-term impact. Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.
if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment [5][6]. These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!
We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants. We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership” [7]. Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms” [7]. It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…
Footnotes:
[*] This paper is now available at https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/53055
Sources:
[1] Simister, N. (2015). Monitoring and Evaluation Series: Outcomes Outputs and Impact. Retrieved from https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Outcomes-Outputs-and-Impact-7.pdf
[2] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved September, 2018, from https://web.archive.org/web/20180919035910/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm
[3] USAID. (2016, December 6). USAID Uganda Country Development Cooperation Strategy 2016-2021. Retrieved October, 2018, from https://www.usaid.gov/uganda/cdcs
[4] Porter, S. (2018, October 18). DAC Criteria: The Hand That Rocks the Cradle. Retrieved from https://views-voices.oxfam.org.uk/2018/10/dac-criteria-the-hand-that-rocks-the-cradle/
[5] European Evaluation Society Biennial Conference: Flagship Symposia. (2018). Retrieved from http://www.ees2018.eu/1539782596-flagship-symposia.htm
[6] Ofir, Z. (2018, October 13). Updating the DAC Criteria, Part 11 (FINAL). From Evaluation Criteria to Design Principles. Retrieved from https://zendaofir.com/dac-criteria-part-11/
[7] OECD. (2018, June 11). 2018 Report On The DAC Untying Recommendation. Retrieved from http://www.oecd.org/dac/financing-sustainable-development/development-finance-standards/DCD-DAC(2018)12-REV2.en.pdf
by Jindra Cekan | May 16, 2018 | Accountability, Aid effectiveness, CGDev, Evaluation, ex-post evaluation, Feed the Future, Feedback Labs, foreign aid, impact investing, Impact Investors, International aid, international development, Local Participants, Participants, Participation, post project evaluation, post-project evaluation, Return on Investment (ROI), SALT, Sidekick Manifesto, SUSRoi, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Sustained and Emerging Impacts Evaluations (SEIE), USAID, Valuing Voices
Setting a higher bar: Sustained Impacts are about All of us
Global development aid has a problem which may already affect impact investing as well.
It is that we think it’s really all about us (individuals, wealthy donors and INGO implementers) not all of us (you, me, and project participants, their partners and governments). It’s also about us for a short time.
All too often, the measurable results we in global development aid and Corporate Social Responsibility (CSR) funded projects that last 1-5 years track and report data for two reasons:
1) Donors have Compliance for grantees to meet (money spent, not lost, and results met by fixed deadlines of 1-5 years – look at some of the European Commission Contracting rules) and
2) Fund recipients and the participants they serve are accountable to ‘our’ donors and implementers who take what happened through their philanthropic grants as ‘their’ results.
Both can skew how sustainably we get to create impacts. An example of such strictures on sustainability from USAID. As respected CGDev Elliot and Dunning researchers found in 2016 when assessing the ‘US Feed the Future Initiative: A New Approach to Food Security?‘ the $10.15 billion leveraged $20 billion from other funders for disbursement over three years (2013-16). “We are concerned that pressure to demonstrate results in the short term may undermine efforts to ensure any impact is sustainable…. Unfortunately, the pressure to show immediate results can encourage pursuit of agricultural investments unlikely to be sustained. For example, a common response to low productivity is to subsidize or facilitate access to improved inputs… it can deliver a quick payoff… however, if the subsidies become too expensive and are eliminated or reduced, fertilizer use and yields often fall…..
With so much focus on reporting early and often about the progress in implementing the initiative, there is a risk that it increases the pressure to disburse quickly and in ways that may not produce sustainable results. For example, for 2014, Feed the Future reports that nearly 7 million farmers applied “improved technologies or management practices as a result of U.S. Government assistance,” but only 1,300 received “long-term agricultural sector productivity.” Are the millions of others that are using improved inputs or management practices because of subsidies likely to have these practices sustained? And how likely are they to continue using improved practices once the project ends?”
3) Impact investors stick to the same two paths-to-results and add a new objective: market-competitive financial returns. They also need to show short-term results to their investors, albeit with social, environmental and governance results like non-profits (future blog).
4) Altruists create things we want ‘beneficiaries’ (our participants) to have. For instance a plethora of apps for refugees cropped up in recent years, over 5,000 it is estimated, which can be appropriate, nor not so helpful. Much like #2 above, ‘we’re’ helping ‘them’ but again, it seems to be a ‘give a man a fish’… and my fish is cool sort of solution… but do our participants want/ need this?
How often is our work-for-change mostly about us/by us/ for us... when ideally it is mostly about ‘them’ (OK, given human self-interest, shouldn’t changes we want at least be about all of us?).
All too often we want to be the solution but really, our ‘grassroots’ clients who are our true customers need to generate their own solution. Best if we listen and we design for long-term sustainability together?
As the Brilliant Sidekick Manifesto stated in two of its ten steps:
a) “I will step out of the spotlight: Sustainable solutions to poverty come from within are bottom-up, and flow from local leaders who are taking the risks of holding their politicians accountable and challenging the status quo.”
b) “I will read “To Hell with Good Intentions” again and again: Politicians, celebrities and billionaire philanthropists will tell me that I can be a hero. I cannot. The poor are not powerless or waiting to be saved. Illich will check my delusions of grandeur.”
We have examples of where we have stepped away and participants had to fend for themselves. At Valuing Voices, we’ve done post project-exit evaluations 2-15 years afterward. What did participants value so much that they sustained it themselves (all about them, literally)? These Sustained and Emerging Impacts Evaluations (SEIE) also give us indications of Sustained ROI (Sustained Return on Investment (SusROI) is a key missing metric. As respected evaluator Ricardo Wilson-Grau said in an email, “I think calculating cost-effectiveness of an intervention’s outcomes would be a wonderful challenge for a financial officer searching for new challenges — if not a Nobel prize in economics!”)
Most of these evaluations are pretty bad news mixed with some good news about what folks could sustain after we left, couldn’t and why not. (These are the ones folks expect to have great results, otherwise they wouldn’t share them!) While most clients are understandably interested in what of ‘theirs’ was still standing, and it was interesting disentangling where the results were attributable by implementation or design or partnership flaws or something else, what was mesmerizing was what came from ‘them’.
The key is looking beyond ‘unexpected’ results to look at emerging impacts that are about ‘them’ (aka what we didn’t expect that was a direct result of our project, e.g. spare parts were no longer available to fix the water well pump once we left or a drought rehabilitation water project that decreased violence against women), to what emerging results are attributable not to use but only to our participants and partners who took over after our projects closed. One example is a Nepalese project ended yet the credit groups of empowered women spawned groups of support groups for battered women. Another is a child maternal health project changed how it worked as women reverted to birthing at home after NGOs left; community leaders punished both parents with incarceration in the health clinic for a week if they didn’t given birth there (wow did that work to sustain behavior change of both parents!).
Many of us at Valuing Voices are shocked that funders don’t seem that interested in this, as this is where they not only take over (viz picture, sustaining the project themselves), but they are making it theirs, not ours. Imagine assuming the point of development is to BE SUSTAINABLE.
Source: Community Life Competence
Our participants and national stakeholder partners are our true clients, yet… Feedback Labs tell us Americans alone gave $358 billion to charities (equivalent to the 2014 GDP of 20 countries) – in 2014 but how much of this was determined by what ‘beneficiaries’ want? Josh Woodard, a development expert, suggests a vouchers approach where our true clients, our participants, who would “purchase services from those competing organizations… [such an] approach to development would enable us all to see what services people actually value and want. And when we asked ourselves what our clients want, we would really mean the individuals in the communities we are in the business of working with and serving. Otherwise we’d be out of business pretty quickly.”
This opens the door to client feedback – imagine if participants could use social media to rate the sustained impacts on them of the projects they benefited from? A customer support expert wrote in Forbes, “Today, every customer has, or feels she has, a vote in how companies do business and treat customers. This is part of a new set of expectations among customers today that will only grow ... you can’t control product ratings, product discussions or much else in the way of reviews, except by providing the best customer experience possible and by being proactive in responding to negative trends that come to the surface in your reviews and ratings stronger.”
So how well are we working with our participants for ‘development’ to be about them?
What do you think?
by Jindra Cekan | Apr 12, 2018 | Accountability, Aid effectiveness, Catholic Relief Services (CRS), Demographics, ex-post evaluation, Feedback Labs, Mercy Corps, PCI, post-project evaluation, SDGs, Sustainability, Sustainable development, Youth, youth employment
Investing in Youth for Project Effectiveness and Sustainability
One out of every six people on earth is between the ages of 15-24, says the UN. That is 1.2 billion youth. As one young leader says, “if the world’s problems are to be solved, it’s not going to happen without us.” Yet in 2015, the International Labor Organization said 73.3 million youth between 15 and 24 were unemployed. Not only do and an estimated 169 million young workers lived on less than $2 a day, 75 percent of youth workers are only informally employed. In Africa alone, the UN estimates 200 million are such youth; not only does Africa have the youngest population in the world, this figure will double by 2045, but the largest numbers remain in Asia (IMF 2015). The World Bank has striking African and Asian demographics:
How often do we fund projects that are designed and run by youth? How engaged are youth in sustaining the projects we have funded, designed, implemented, monitored and evaluated? What have we in global development, including corporate social responsibility and investing spheres done to ensure that youth are both engaged in our projects, but are in the leadership to direct how they are done now, and sustaining them beyond donor departure? Further, how well are we collaboratively developing technology with them for them to use to thrive in this sped-up, high-tech world?
Valuing Voices youth blogs covered the barriers to youth success in the ‘developing world’ which included a lack of access to sufficient numbers of jobs, compounded by a lack of job-appropriate skills, access to capital, decision-making etc. We heartily agree with the IMF that “youth have a huge stake in bringing about a political and economic system that heeds their aspirations, addresses their need for a decent standard of living, and offers them hope for the future…. [Also] that communities, cities, provinces, and countries can set up forums for the purpose of listening to the concerns and ideas of adolescents and young adults and stimulating change. Young people could be offered a voice in decision-making bodies…Inclusion can benefit all.”
Why should we make this happen? Taking inclusive steps fosters sustained impacts long after we grow old.
As CRS Niger’s otherwise very successful Sustained and Emerging Impacts Evaluation of the food security project shows us, there is much room to grow in inclusion of ‘youth’ (up to age 35 in Niger), both by including youth:
- The exodus of youth diminished during and after the project [by] using the same land to train 100 new vegetable farmers and trainers. Youth seasonal outmigration decreased due to increased food production, especially due to vegetable gardening even during the dry season, and increased knowledge of practices such as rainfed agricultural [practices] which kept youth locally employed.“
- Youth, too, having learned [agricultural water-conservation] techniques, and generated income [even] while seasonally working outside the village.
Versus not engaging youth:
- Although most committees are still functioning, there are no processes in place to engage and train youth and new inhabitants of the villages [in project activities after close-out]… … there are serious questions about how well they will be engaged and train youth and new members of the communities and how much will be transferred cross-generationally. This is pivotal given that 50% of Nigeriens are under the age of 15;
- The [sustainability] problem was that [youth] were not elected or chosen for the [management] committees. This is another issue to flag in other projects interested in sustainability: the implications of selecting a limited number of elders to staff multiple committees than a broad array of young committee members that could grow into leadership positions. Given the youth’s overall dissatisfaction with group leadership, other projects need to be aware of “elite capture” and its potential threat to sustainability
Investing in youth is a terrific investment in sustainability. How often do we consider it?
A post-project example from Mercy Corps/ PCI’s early ex-post evaluation in Central Asia showed that such investments are not easy but can pay off after the project closed. “72% of youth report that they continue to use at least one skill they learned during the [infrastructure] program”… including teamwork and communication, sewing, construction, roofing, journalism and cooking.” This may have been in part due to the project’s youth summer camps, organized each year to promote youth leadership and participation in community decision-making, which were supported by [some of] the adult population. While the project “encouraged communities to elect young people as representatives…within the cultural context this was not met enthusiastically by the communities because young people were not felt to be ‘qualified’ as leaders.” Yet inter-generational collaboration was fostered by the project by establishing mentoring programs where older people with technical skills mentored [some] young people during the infrastructural construction activities.”
Raj Kumar, Founder of Devex and chair for the World Economic Forum said this about what to do post-Davos: “With a dozen years to go before the finish line of the Sustainable Development Goals, we need to get the underlying plumbing right in order to have a chance to reach those goals. That plumbing includes everything from having the country-level data to track progress against the goals to having the project-level data to know what’s working and what’s not…. Most importantly, it’s about the development leaders of today building out the best systems so the development leaders of tomorrow can focus on delivery.”
What we at Valuing Voices are most encouraged by is the prospect of overtly considering sustained impacts to inform the funding, designing, implementing, monitoring and evaluating of projects today for the adult millennials of tomorrow. Regarding youth, we will need a mix of focused initiatives such as longstanding work by the International Youth Foundation, and new investments funds such as the Global Youth Empowerment Fund and integrating youth into projects at all stages of the projects and beyond, as we showed regarding CRS and Mercy Corps/PCI, above.
That is one way to get SusROI (Sustainable Return on Investment).
As mother who has worked in 26 countries, I feel the great urge to harness youths’ yearning to succeed through their love of technology. The growth of mobile money in Africa is one example of technology use in daily life. Who is supporting youth employment in technology? Mercy Corps’ 2017 Social Ventures Fund that supports “positive trends offered by technology. Trends in micro-work, micro-manufacturing, digital livelihoods and mobile-enabled agent networks are paving the way for the acceleration of a distributed and digitally enabled workforce and the reinvention of manufacturing, sales and distribution. Their investments related to youth employment are:
- NewLight Africa – network of rural sales and customer service agents
- Wobe – anyone with an Android phone in Indonesia can be a micro-entrepreneur
- Lynk – job-matching platform
- Sokowatch – network of urban sales and customer service agents
I dream of youth crowd-sourcing post-project sustained impact results. Feedback Labs has a lot of really interesting tools for… feedback from people on the ground in-country. I searched high and low and found only this crowdsourcing data collection overview of three aid research tools for ICT4D (information and communications technology for development), the best of which appears to be Findyr. (FYI here is feedback on the limitations of crowdsourcing in emergencies.)
Also, the ‘impact tracking’ platform by Makerble looked good but who among you have a wider perspective to advise us what’s best?
Finding technologies and funding to hear youth’s voices and feedback on what they could sustain or could not after our projects closed, and why is unbelievably valuable to inform funding, design, implementation, M&E, and of course foster youth empowerment. Good listening to participants comes first. As an impact evaluation in Uganda found that “when villagers and teachers, instead of school officials, are allowed to set their own priorities for improving schools and directly monitor performance, the results can be priceless. In Uganda, World Vision knew that community-based monitoring of school performance could help sustain improvements in education that building schools, supplying textbooks, and training teachers alone could not. They tried two approaches: the use of a standard scorecard with performance questions identified by education officials and development partners, and a participatory scorecard, where community members defined the issues they would monitor. A randomized controlled trial [RCT] revealed that the participatory scorecard delivered more than the standard scorecards. The participatory approach prompted higher efforts by teachers, as expected. But it also prompted higher efforts from villagers— local politicians learned more about their country’s education policies and what they could advocate for on behalf of their constituents, parents increased their support of schools by contributing to midday meals, and children found a forum to report teacher absenteeism and other factors that hurt their education. In the end, while the standard scorecard made little difference in school performance, the participatory approach improved attendance by teachers and students and helped raise student test scores.’”
By accessing mobile technology, ground-truthing project sustainability, given youth’s familiarity with technology and network-interconnected habits, I believe together we can cost-effectively democratize evaluations and help ‘development’ be ‘sustainable’. Collaborate with us!
by Jindra Cekan | Nov 29, 2017 | Aid effectiveness, Better Evaluation, ex-post evaluation, Honduras, Madagascar, Niger, Nigeria, post-project evaluation, Sustainable development, Sustained and Emerging Impacts Evaluation, USAID
Can’t wait to learn from post-project sustainability evaluation? If not why?
A colleague who has been promoting ex-post sustainability evaluation in her organization questioned my claim that doing them had “benefits” for future programming. It was an “untested assumption that there will be sufficient, strong enough evidence to apply to future programming… [and] the need to have evidence to cite for future work is not pressing enough.”
If you are on aid’s receiving end, what you care about is that good results are sustained, and you are able to live better, longer. You might want to show others evidence of what was sustained, rather than only what worked while external investments were there but stopped since. Absolutely, aid donors need to have evidence that something designed, funded, implemented, and monitored & evaluated showed good results, but we assume our results will be sustained after we have closed out and moved on. How well have we done? Let’s see.
At the American Evaluation Association meetings this month, several post-project evaluations were presented. Some came from Valuing Voices research, some from Social Impact, PLAN and World Vision and some others [1].
Results 3-5 years post close-out were, shall we say unexpected, from CRS Madagascar:
[2]
Nigeria,
[3]
and Honduras.
[4]
While there were some successes, including Niger
[5]
and Burkina Faso, where MCC/ PLAN found that three years post project “BRIGHT still had a significant positive impact—6.0 percentage points for children between ages 6 and 22—on self-reported enrollment. The impacts are smaller than estimated impacts on enrollment at 7 and 3 years after the start of the program,” they were rare [6].
If we don’t wonder why things didn’t work or why they did, and don’t return to find out if it happened again and what to do/ not to do again?, Often we continue to do very similar programming elsewhere, again assuming great results. How can we close our eyes and not do post project, Sustained and Emerging Impacts Evaluations (SEIE) and see, learn, do better? How can we continue to do very similar water/ sanitation, health, food security, and education programs and projects (with potentially similar results), and call ourselves sustainable development professionals? Shouldn’t we always ask not how effective is our aid when it’s there, but after its gone?
If you want more data, see a presentation we did at USAID. What do you think?
Sources:
[1] Cekan, J., PhD. (2014, April 7). Evaluation of ERCS/Tigray’s “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity”. Retrieved from https://valuingvoices.com/wp-content/uploads/2017/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
[2] Madagascar Rural Access To New Opportunities For Health And Prosperity (RANO-HP) Ex-Post Evaluation. (2017, June 1). USAID. Retrieved from https://www.globalwaters.org/resources/assets/madagascar-rural-access-new-opportunities-health-and-prosperity-rano-hp-ex-post-0
[3] The World Bank. (2014, June 26). Project Performance Assessment Report, Nigeria: Second National Fadama Development Project. Retrieved from http://ieg.worldbankgroup.org/sites/default/files/Data/reports/Nigeria_Fadama2_PPAR_889580PPAR0P060IC0disclosed07070140_0.pdf
[4] Rogers, B. L., Sanchez, L., & Fierstein, J. (n.d.). Exit Strategies Study: Honduras. Retrieved from https://www.fsnnetwork.org/sites/default/files/fanta_exit_strategies_presentation_honduras_-_final_-_feb_5_0.pdf
[5] Cekan, J., PhD, Kagendo, R., & Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all
[6] Davis, M., Ingwersen, N., & Kazianga, H. (2016, August 29). Ten-Year Impacts of Burkina Faso’s BRIGHT Program. Retrieved from https://www.mathematica.org/our-publications-and-findings/publications/ten-year-impacts-of-burkina-fasos-bright-program
