by valuingvoicesjin | Jul 25, 2014 | Aid effectiveness, Ethiopia, Evaluation, Feedback Labs, Feedback loops, Food security, LIvelihoods, Participants, Red Cross, Sustainability
Reposted from Feedback Labs: http://feedbacklabs.org/see-how-it-turned-out-feedback-loops-for-implementation-and-sustainability/
When I met Aminata in a central Malian village, she asked me whether I was with the people with the yellow trucks or the white trucks. That was her way of differentiating between development projects. I explained to her I was doing (PhD) research, with neither. She asked me whether they (the trucks) would come back. I couldn’t answer, and therein lies the problem. “Country-led development” begins with communities being involved at every stage of a project. Ongoing community input during project design, implementation, and monitoring is needed for impact, local community ownership and sustainability. Developing “feedback loops” that facilitate two-way communication are key for building cultures of collaboration, adaptation and learning into international development programs. Valuing Voices sees data as another resource to deliver development, beyond serving the needs of donors and international non-profits.
The distance between our intentions and our reality
Too often, data is extracted from communities by development organizations, in order to evaluate how well they fulfilled the project, rather than communitiesevaluating how well international development projects supported community needs. The best projects co-design interventions and monitoring with communities. Too often communities have no mechanism to learn how their feedback during implementation or during evaluations led to real changes. Such feedback leads to community buy-in, as there is proof that their voices matter and that they are co-driving development. Outside of submitting a format report in a PDF to a donor, the development field has consistently inadequately retained the data from projects once they close, much less leave it in-country, and rarely return afterwards for follow-up evaluations. This further reinforces the idea that these communities are being exploited for resources or development experimentation, with no thought to long-term capacity, learning or sustainability.
Ever farther from this vision of collaborative partnership in project conceptualization, design, and monitoring is what happens once these short and long term projects are closed out. Due to budget restrictions and bureaucratic habits, too often the task of sustaining these projects is handed over to local partners, without funding, staff or data continuity. Cursed with “pilot-itis”, development initiatives too often lose sight of achieving scalability and sustainability. ‘Sustainability’ is often incorrectly defined only as whether the specific project got follow-on funding, rather than by asking communities what activities they were able to continue long after the project ends. Communities provide feedback only 1% of the time, missing great learning opportunities to be had from returning to assess these same projects 1-10 years later to see what was expected, unexpected and what could be sustained. We therefore never truly learn what had the greatest prospect for replicability and scaling elsewhere.
How do we really show that we Value Voices?
At Valuing Voices, we believe that community members are our true clients. We have identified two kinds of feedback loops that are needed to make international development far more effective.
-
Country-Led Project Implementation Feedback Loops: Valuing Voices wants to create standardized methodologies and data collection processes that can be integrated into most international development projects to create feedback loops that continue working long after projects close. Valuing Voices believes key elements to developing collaborative feedback loops are:
-
Identification of different feedback loop methodologies. Based on what is appropriate for the population, Valuing Voices identifies different methodologies to create these loops through a mix of traditional and digital tools. This means explicitly targeting the building, capturing and sharing of feedback during a specific project, to test and document different methodologies and create standard processes and infrastructure. This includes participatory methodologies and studies such as Empowerment Evaluation and “Who Counts? The Power of Participatory Statistics, as well as ways of evaluating qualitativeStorytelling. We must of course protect our respondents as well.
-
Use of a franchise approach to replicate and scale. The identified methodologies can be taught to local and in-country evaluators and development experts who can then “sell” those services to governments, local NGOs, international donors, and the private sector. Valuing Voices is a catalyst for this country-based franchise approach which strengthens national evaluation, capture and learning. It also allows for feedback loops to exist within a program as local evaluators provide feedback on impact and lessons for improvement. This leads to local empowerment, sustainability and aid transparency.
-
Use of digital tools- capture once, share forever. Following evaluations on the local level, Valuing Voices then rolls up national evaluations by sector, analyzes them for what’s most sustained, and shares that learning around the world, influencing project design, implementation, funding and empowerment. In addition to generating feedback loops, this valuable feedback is entered into a curated database of findings for comparison across projects such as agriculture, livelihoods, credit, natural resources, health and education. By using structured data analysis, we can compare feedback data and actual behavior. We will look at whether we are capturing the right associated metadata (date, location, project) to contextualize feedback and bring forth lessons our partners can learn from, analyzing similarities and learning from them to improve program quality, organizational learning (of all partners) and sustainability prospects.
-
Post-Project evaluation Feedback Loops: The eagerly awaited Local Systems: A Framework for Supporting Sustainable Development was published by USAID in May:
“More ‘ex-post’ evaluations — which are meant to determine the impact of a project after it is completed, sometimes years later… are needed to design and implement projects.”
This is great news, as in the last 24 years, USAID has not published a single post-project evaluation, and the World Bank has only published one. This could indicate that none were written, or none were seen as ‘publishable’.
Post-project (ex-post) evaluations should look at:
-
The resilience of expected impacts of the project two, five, and ten years after close-out;
-
The communities’ and NGOs’ ability to self-sustain those activities;
-
Positive and negative unintended impacts of the project, both immediately in in the long term;
-
Which activities the community and NGOs felt were successes which could not be maintained without further funding;
-
Lessons for other projects on what interventions were most resilient which communities valued enough to continue themselves or for which NGOs valued enough to obtain additional funding, as well as what was not resilient.
These evaluations are rarer than snow in Sri Lanka. Feedback loops post-implementation are key to understanding the sustainability of projects and to improve transparency, efficacy and learning. Valuing Voices has a handful of examples of ex-post (post-project) evaluation done by organizations such as Mercy Corps and Plan International as well as bilateral JICA, but shockingly 99.9% of all development projects just don’t go back and check.
Voices to be valued
These are just a few examples of areas where we can value the voices of our communities, to be more effective and impactful with our efforts, and show our respect and value of communities’ input.
The time is ripe for growth, for these voices to be heard more loudly, telling us what they want development to be. Join us in advocating for funding for such feedback, join us as partners in our field sites, as partners in ICT database creation, as voices for communities.
Communities want their voices heard. Sustained development depends on it.
Jindra Cekan, PhD is the founder of Valuing Voices at Cekan Consulting LLC. She has worked in international development for 25 years in participatory design, monitoring & evaluation and knowledge management in over 20 countries. Her PhD was in Mali, “Listening to One’s Clients” and she consults to non-profit, public (USAID), foundation and private sectors.
Siobhan Green, MA is the founder of Sonjara, Inc, and a member of ValuingVoices. She has worked in international development since 1992, and specializes in ICTs for development, knowledge management, and technology for monitoring and evaluation. Her master’s thesis was on “The Internet in Africa: Policy Perspectives and Approaches” in 1997, and she works with USAID and other USG clients, non-profits, and for-profit partners.
by valuingvoicesjin | May 9, 2014 | Africa, LIvelihoods, Participation, Unemployment, Youth
Youth Series Part I: Factors Hindering Youth Participation in Development
My most recent research has been focused on global youth and their capacity (or incapacity) to be integrally involved in their own development process. With such huge youth cohorts in developing countries, collectively referred to under the term “youth bulge”, there is no doubt about whether the power of this demographic should be harnessed to achieve development goals, but rather how. This first installation of youth blogging will thus focus on the question: what are some of the key factors that hinder youth advancement and participation in their societies? In other words, what is holding youth in development back?
Here are some key facts:
- Roughly 85 percent of global youth live in developing countries, with half living in low-income countries.
- Around 238 million of these youth are living in positions of extreme poverty, surviving on less than one dollar a day [1]. No where is poverty felt more extremely than in Africa, because despite a sharp reduction in global poverty over the past thirty years, this percentage has not significantly fallen in Africa, particularly in Sub-Saharan Africa where, “over 40% of people…[still] live in absolute poverty” [2].
- North Africa especially, has the highest (exceeding 25 per cent) youth unemployment in the world — significantly higher than the 17.3 per cent of the OECD area. Indeed, North Africa (with the Middle East) is the only region where youth unemployment exceeds 20 per cent globally, showing that unemployment is particularly acute among the youth (15–24 years) [3].
The overall absorptive capacity of African economies is weak, which has heavy costs for the increasingly desperate youth cohorts who are unable to find suitable work.
- In Tanzania, about 800,000 people enter the labor market each year,” yet the government is only able to absorb 40,000 or only about 5 percent of the newly available workforce.
- In places like Egypt, the majority of new jobs being created are either poor-quality or low productive waged jobs [4]. This is very discouraging for the youth there who have the highest levels of educational attainment as compared to all the past generations.
- Even among some of the fasted growing economies in Africa, such as Ethiopia, which has annual growth averaging above 10% the youth demographic in particular faces formidable employment prospects especially in urban areas where it is estimated to be 20.6%.
Across the entire continent of Africa, “youth are faced with bleaker life prospects and are disenchanted with policies and established institutions for failing to provide them with opportunities to fully reach their potential and to live in dignity” [5]. The result of this disenchantment has dismal repercussions, such as in Zimbabwe, where, “young people are losing interest in being educated because wages and salaries are low an unattractive” and jobs are so hard to find [5]. The consequences of youth choosing not to receive an education, because even if they do they can’t even acquire a good job in today’s market, poses a threat to the ongoing social development and poverty reduction of nations.
Further, the scarcity of quality job opportunities for youth is credited to the lack of preparedness for youth transitioning from school into the workplace. “In Zambia and Tanzania, young people attributed the skills mismatch currently faced in the labor market to the poor flow of information regarding skills demanded by potential employers” [5]. In other words, without a system in place to facilitate in-demand skills training that takes into account what employers are actually seeking, youth are unable to adequately prepare themselves for the realities of the job market they wish to enter by getting relevant skills training. This is due to the systematic lack of communication between the key institutions that youth interact with, including universities, vocational schools and skills development/training institutions, because “schools and universities [in Africa] provide mass education rather than quality service. There is a general deterioration of infrastructure, and a lack of collaboration between the educational system and potential employers, as well as poor accessibility of training services, in many countries in the region” [5]. Without competitive, up-to-date skills training, these institutions are perpetuating the existing challenges that the un/underemployed youth face, making their levels of preparedness insufficient for the market and further contributing to the circumstances of poverty that keep youth disenfranchised and unable to participate to their fullest potential in the socioeconomic development of the region.
International development needs to align itself with the issues that developing youths most directly face – lack of skills and jobs, disempowerment and disillusionment. With this basis in understanding about the inhibiting factors that disempower youth in developing countries, stay tuned for Part II of this analysis to see which factors actually serve to enable youth participation, and what some potential solutions could be.
Sources:
[1] Advocates for Youth. (2005). Youth and the State of the World. Retrieved from https://www.advocatesforyouth.org/wp-content/uploads/storage//advfy/documents/fsstateworld.pdf
[2] Our Africa. (n.d.). Poverty. Retrieved February 6, 2014, from https://web.archive.org/web/20140206110858/www.our-africa.org/poverty
[3] Anyanwu, J. C. (2013). Characteristics and Macroeconomic Determinants of Youth Employment in Africa. African Development Review, 25(2). Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8268.2013.12019.x
[4] Barsoum, G., Ramadan, M., & Mostafa, M. (2014, June). Labour market transitions of young women and men in Egypt. Retrieved from https://www.ilo.org/employment/areas/youth-employment/work-for-youth/publications/national-reports/WCMS_247596
[5] International Labour Organization (ILO). (2012, June). Africa’s Response to the Youth Employment Crisis. Retrieved from https://www.ilo.org/africa/WCMS_184325/lang–en/index.htm
by valuingvoicesjin | May 1, 2014 | Accountability, Asia, Bilateral organizations, Donors, Evaluation, impact evaluation, International aid, International non-profits, Local Participants, NGO, Participation, Resilience, Results, Sustainability, Sustainable development, USAID, water
Pineapple, Apple- what differentiates Impact from Sustainability Evaluation?
There is great news. Impact Evaluation is getting attention and being funded to do excellent research, such as by the International Initiative for Impact Evaluation (3ie), by donors such as the World Bank, USAID, UKAid, the Bill and Melinda Gates Foundation in countries around the world. Better Evaluation tell us that “USAID, for example, uses the following definition: “Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other that the intervention that might account for the observed change.”
William Savedoff of CGD reports in Evaluation Gap reports that whole countries are setting up such evaluation institutes: “Germany’s new independent evaluation institute for the country’s development policies, based in Bonn, is a year old. DEval has a mandate that looks similar to Britain’s Independent Commission for Aid Impact (discussed in a previous newsletter ) because it will not only conduct its own evaluations but also help the Federal Parliament monitor the effectiveness of international assistance programs and policies. DEval’s 2013-2015 work program is ambitious and wide – ranging from specific studies of health programs in Rwanda to overviews of microfinance and studies regarding mitigation of climate change and aid for trade.” There is even a huge compendium of impact evaluation databases.
There is definitely a key place for impact evaluations in analyzing which activities are likely to have the most statistically significant (which means definitive change) impact. One such study in Papua New Guinea found SMS (mobile text) inclusion in teaching made a significant difference in student test scores compared to the non-participating ‘control group’ who did not get the SMS (texts). Another study, the Tuungane I evaluation by a group of Columbia University scholars showed clearly that an International Rescue Committee program on community-level reconstruction did not change participant behaviors. The study was as well designed as an RCT can be, and its conclusions are very convincing. But as the authors note, we don’t actually know why the intervention failed. To find that out, we need the kind of thick descriptive qualitative data that only a mixed methods study can provide.
Economist Kremer from Harvard says ““The vast majority of development projects are not subject to any evaluation of this type, but I’d argue the number should at least be greater than it is now.” Impact evaluations use ‘randomized control trials’, comparing the group that got project assistance to a similar group that didn’t to gauge the change. A recent article that talks about treating poverty as a science experiment says “nongovernmental organizations and governments have been slow to adopt the idea of testing programs to help the poor in this way. But proponents of randomization—“randomistas,” as they’re sometimes called—argue that many programs meant to help the poor are being implemented without sufficient evidence that they’re helping, or even not hurting.” However we get there, we want to know – the real (or at least likely)- impact of our programming, helping us focus funds wisely.
Data gleaned from impact evaluations is excellent information to have before design and during implementation. While impact evaluations are a thorough addition to the evaluation field, experts recommend they be done from the beginning of implementation. While they ask “Are impacts likely to be sustainable?”, and “to what extent did the impacts match the needs of the intended beneficiaries?” and importantly “did participants/key informants believe the intervention had made a difference?” they focus only on possible sustainability, using indicators we expect to see at project end rather than tangible proof of sustainability of the activities and impacts that communities define themselves that we actually return to measure 2-10 years later.
That is the role for something that has rarely been used in 30 years – for post-project (ex-post) evaluations looking at:
- The resilience of expected impacts of the project 2, 5, 10 years after close-out
- The communities’ and NGOs’ ability to sustain which activities themselves
- Positive and negative unintended impacts of the project, especially 2 years after, while still in clear living memory
- Kinds of activities the community and NGOs felt were successes which could not be maintained without further funding
- Lessons for other projects across projects on what was most resilient that communities valued enough to do themselves or NGOs valued enough to get other funding for, as well as what was not resilient.
Where is this systematically happening already? There are our catalysts ex-post evaluation organizations, drawing on communities’ wisdom. Here and there there are other glimpses of ValuingVoices, mainly to inform current programming, such as these two interesting approaches:
- Vijayendra Rao describes how a social observatory approach to monitoring and evaluation in India’s self-help groups leads to “Learning by Doing”– drawing on material from the book Localizing Development: Does Participation Work? The examples show how groups are creating faster feedback loops with more useful information by incorporating approaches commonly used in impact evaluations. Rao writes: “The aim is to balance long-term learning with quick turnaround studies that can inform everyday decision-making.”
- Ned Breslin, CEO of Water For People talks about “Rethinking Social Entrepreneurism: Moving from Bland Rhetoric to Impact (Assessment)”. His new water and sanitation program, Everyone Forever, does not focus on the inputs and outputs, including water provided or girls returning to school. Instead it centers instead on attaining the ideal vision of what a community would look like with improved water and sanitation, and working to achieve that goal. Instead of working on fundraising only, Breslin wants to redefine the meaning of success as a world in which everyone has access to clean water.
We need a combination. We need to know how good our programming is now through rigorous randomized control trials, and we need to ask communities and NGOs how sustainable the impacts are. Remember, 99% of all development projects worth hundreds of millions of dollars a year are not currently evaluated for long-term sustainability by their ultimate consumers, the communities they were designed to help.
We need an Institute of Sustainable Evaluation and a Ministry of Sustainable Development in every emerging nation, funded by donors who support national learning to shape international assistance. We need a sustainability global database, mandatory to be referred to in all future project planning. We need to care enough about the well-being of our true client to listen, learn and act.
by valuingvoicesjin | Jan 15, 2014 | Accountability, Africa, Bilateral organizations, Governance, International aid, International non-profits, LIvelihoods, Local Participants, Results, Sustainability
Let's start turning the oceanliner of development to dignified sustainability – today!
No time like the present, our participants are waiting for dignified development to fully arrive. Dignity is the "quality or state of being worthy of esteem or respect." When we design projects with communities with long-term self-sustainability as the core value, we are respecting them.
And we must start local. As Acumen Fellow Natalie Grillon says from Uganda: "Ask more questions, listen and learn. I’m always trying to get better at listening to learn before I act so that my actions can lead to productive results based on consensus and conversations rather than assumptions. Don’t believe that you know the full story. The farmers for whom I work know their crop, know their land and know what they want."
She goes on to tell us how she stopped doing things for the project staff: " I could offer more in working alongside my colleagues to learn how to do new things together, like using a new database or thinking about ways to motivate teams… I could feel the sense of empowerment and excitement that could come from learning even a simple new thing and which only required a few minutes of our time" rather than just doing it herself for them.
This is what Valuing Voices is all about — seeing our participants as our true clients, as listening to what they want rather than designing development requests for proposal and even project proposals themselves in capital cities far from input by these very clients. Would any corporation in their right mind design without asking potential clients, without pre-testing and tweaking interventions before a full launch? Imagine how products would fare without marketing campaigns touting their benefits over other products? This doesn't happen often in development because participants are most often seen as 'price takers', grateful for what we offer, not matter if the fit is not perfect. Such an approach is not only inefficient, it is deeply disrespectful.
On the other hand, there are successes to be seen along the lines of localizing development. In addition to successes found in my other blogs (see Plan, Mercy Corps, LWR and even PACT – forthcoming), Devex recently posted an article praising Nuru and Millenium Villages for "working to develop local leaders who will fully take over the various programs they’ve begun after the foreigners leave in two years… [using] the opportunity to build up local leaders and engage them in developing their own communities." Hallelujah! Devex says Munk's critique of Millenium Villages is that "Millennium violates two basic principles of good development: It’s not scalable and it’s not sustainable". Nuru believes "poor people hold the keys to their own development."
Absolutely! Not only should design and implementation center around community wants and capacities, there are all sorts of project activities that communities can help sustain themselves:
* Agriculture and livelihoods (income generation, micro-credit, marketing),
* Natural resource management (climate-smart agriculture, reforestation),
* Literacy and numeracy.
While there are other things larger than communities that need ongoing external support for, how often are there referendums on what they would prioritize? The UN recently named 2014-2024 the sustainable energy decade and infrastructure such as roads, water systems as well as trade including the World Trade Organization's Doha Trade talks for improve the trading prospects of developing countries are vital, how often have citizens been asked? Ashoka's Changemakers supports projects that create feedback loops (like our own in East Africa: http://www.changemakers.com/project/valuing-voices-kenya).
Communities are deeply grateful for assistance yet they want to to have a voice, to steer the ship more themselves. The Listening Project found that "agencies should slow down and take more time to understand people’s capabilities, priorities, preferences, and ideas… [participants] don’t want to have aid agencies to be more extractive in how they gather information. They want to be part of the decision making process of aid efforts. This goes beyond two-way communication and requires rethinking many of our assumptions and processes to find ways to truly collaborate and support those who are affected."
Among the project's 6,000 interviewees, some wondered "why no one seems to check on whether the assistance provided has made a positive difference in recipients’ lives. It is important for aid agencies to have processes and mechanisms to receive and provide feedback to communities and to be accountable for their actions—and particularly for any harm that has been done."
Yes! We need to help those projects such as Nuru whose farmers are supported in self-sufficiency, countries that support communities evaluating our projects, for a start. That is the dignified life path to take as a development professional – as their peers.
by valuingvoicesjin | Jan 6, 2014 | Accountability, Bilateral organizations, DIFD, Evaluation, International aid, International non-profits, LIvelihoods, Participation, Resilience, Results, Sustainability, USAID
Development= A Jeep (motor optional).. Resilience? If within 5 years!
Imagine being given a lovely new Jeep. You get a driver (remember driving school) to help you learn to steer it around the pothole-strewn, scantly lit roads. Eventually you take over the controls of the Jeep and control the steering wheel directly, driving offroad, with the copilot praising your good driving and steering only to avoid catastrophe. You are told that one day the Jeep will be yours.
That day arrives. The development agency hands you the keys to the Jeep. You wave good bye to them, return to the Jeep, turn the key. Dead.
Looking under the hood, you realize the motor is gone. Checking the rest of the Jeep you realize there is no fuel and the tries are flat. That is what it is like from the community's view of development projects after close-out. The local NGO to whom the project has been 'handed over' has scant financial or human resources to continue (no engine), and in the last few months' scramble to close out, the implemeneting agency put in few systems for communities to continue doing the programming without support by the local NGO or all the resources they had poured in (no fuel). There is little to help you move the Jeep (even on flat tires) except your own feet, other than the capacity building that was learned early on, as it wasn't built to last based on local materials. Sustainability isn't programmed in projects that have set timelines and donor-set markers of success which mandate close-out.
So you own the Jeep but with little power to move, very much like countless well-meant tractor for development agriculture before you.
There are several glimmers of hope. What communities have is the human power that exists locally, fuelled by participation coupled with information transmission such as WorkWithUs and MakingAllVoicesCount (based on the moral imperative of it's Their Development as well) and ALNAP's push to use evaluation for learning in international development.
We could also help donors springboard to learn from sustainability via the latest buzzword in development: "Resilience". At USAID and DIFD this is the "ability of people, households, communities, countries, and systems to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth" and "helping communities and countries to be better prepared to withstand and rapidly recover from a shock", respectively.
Resilience could be the doorway to getting community-defined sustainable programming to break the cycle of recurrent emergencies that divert resources from long-term development. Imagine: we could ask citizens what will make them resilient! A rare, shining example is a USAID-funded Ethiopia project with a mandate to use participatory impact assessments (process monitoring plus participatory input to capture local perceptions of benefits) to learn from communities. A USAID Solicitation tells us "seventeen impact assessments on different program activities were undertaken to inform best practice and to develop guidelines and policies. A major impact was the development and adoption of Emergency Livestock Guidelines by the Ethiopian government. These were based on best practice assessments in many countries (including Kenya) and action research on different types of interventions. Emergency de-stocking–selling livestock early in a drought to preserve their price and leave more fodder and water for remaining animals — was found to be particularly effective, with a 40:1 benefit cost ratio. Emergency livestock vaccination campaigns, on the other hand, were found to have no impact on livestock mortality, and were dropped in favor of other health interventions including parasite control and de-stocking."
Excellent Valuing of pastoralist Voices! How are such locally-informed excellent processes and findings being widely shared and implemented? What do you think?
