by Jindra Cekan | Feb 13, 2015 | Accountability, Aid effectiveness, Bilateral organizations, Czech, Donor Driven Development, Donors, Evidence-based policy, ex-post evaluation, International aid, NGO, post-project evaluation, Republic of Georgia, self-sustainability, Sustainable development
Czech it out! Great evaluation happening in the Czech Republic
One of the delights of living in another country is the surprises one encounters. For me, coming back to our second 'home', it was an evaluative surprise. For by connecting to the Czech Foreign Ministry's Evaluation team, I found evidence of learning from meta-evaluation, doing ex-post evaluation, conscientious tracking of project cost-effectiveness and an openness to self-sustainability research funding and using national evaluators to lead them.
Czech Foreign aid is widespread- "Through development cooperation, the Czech Republic helps to eradicate poverty in less developed parts of the world by means of sustainable socio-economic development. It also contributes to global security and stability, conflict prevention, the promotion of democracy, human rights and fundamental freedoms, and the rule of law". Development assistance is done by several entities, the main two under the Ministry of Foreign Affairs, ORS (Development Cooperation and Humanitarian Aid) and its subsidiary CRA (Czech Development Agency).
The Ministry of Foreign Affairs oversees some fascinating evaluation work. After attending several partner-donor meetings and presentations of a meta-evaluation, an ex-post from an array of projects in Georgia and a discussion of findings across all evaluations in 2014, I am impressed. Why? Because not only are they willing to learn from both successes and failures, openly discussing challenges in learning between grants and contracts, but also because they are tackling programming in 10 countries (e.g. Afghanistan, Bosnia and Herzegovina, Ethiopia, Georgia) with a mere 17 people and a budget of $35 million for 2015.
What are some of the things they are learning?
1. They commissioned a meta-evaluation looking at 20 projects from 2012-2013. What worked well was well described and documented evaluations that were also cost-effective (evaluations were 4% of total costs) and tried to offer constructive solutions to things that did not work well in projects. While some methodologies needed to be better, and reports were hard to access, a major finding was what needs to be improved is inclusion of local recipients in stakeholder analyses, soliciting their views on what the evaluations should focus and on how the projects affected them. Further, during discussions we highlighted the need for an evaluation of outcomes and impacts, not just how evaluations quality was but also which organizations had the best results and why.
2. They commissioned an ex-post evaluation across eight organizations' in the Republic of Georgia (5 Czech, 3 Georgian), of one-year projects with 131 separate activities in civic engagement, media and youth between 2008-2012. The evaluation looked at the short-term effectiveness and longer-term sustainability of activities in the Republic of Georgia. Key findings included good relevance of aid offered, high cost-efficiency, low effectiveness for Georgian decision-making, primarily individual (rather than systemic) sustainability, though some good impact.
Key recommendations from this evaluation-, which Valuing Voices thinks, are universal included:
LEARN BETTER TOGETHER
a. Implement min. 3-year projects, whereby focus in a selected region (or a few regions) on a selected local priority topic, ensure in-depth needs analysis, multi-stakeholder cooperation [including participants], sustainable mechanisms, ongoing local support and enough flexibility as per external factors.
b. Allocate budget for burning human rights issues and for enhancing planning, monitoring, evaluation and learning capacities of Civil Society Organizations.
and
SHARE FINDINGS MORE
c. Coordinate activities with other implementers and donors in the target area and if possible (taking into account the political situation) also with local state institutions and potential implementers
d. Implement multi-stakeholder initiatives in a specific area (health, environment, social inclusion, minorities) with an advocacy component, sharing of results / lessons learnt and a media component
3. Among annual recommendations from Evaluation discussions throughout the year emerged this surprising one on cost-effectiveness. A detailed financial budget is now standard, and expenses for project activities among a majority of (grant-funded) projects and the Czech Development Agency are now required. This enables cost-effectiveness comparisons at least across grants (albeing not across for-profit contracts). In my experience this is unparalleled! (Let me know if other countries do this please!)
Overall, the fact that the Czech Foreign Ministry and implementing partners are willing to look at themselves critically and transparently improve accountability to its ultimate recipients and taxpayers makes me shout Hurrah from all of Prague’s 100 spires! Here is one of them, taken from a Ministry’s window.
by Jindra Cekan | Dec 16, 2014 | Evaluation, India, Industrial Revolution, International aid, Shrujan, South Africa, Sustainable development, Umzi Wethu, Valuing Voices
Ideas we’re not ready for are the ones we need most
Guest Post: Barbara Geary Truan
Some ideas precede and shape reality, the ones that lie behind everything we create. Other ideas lag behind an often hidden reality: they seek then find it. Galileo Galilei observed the skies for years before coming up with the notion that the earth was revolving around the sun. Both kinds of ideas have something essential to offer development professionals today.
The first tells us that if we can envision a world without poverty, we can create it. But it also begs the question of why, in spite of the last 50 years of development work, more than a billion people today live on less than $1.25, putting them below the internationally accepted norm of extreme poverty. This, I think, is where the second kind of idea comes in.
If we gaze at the development skies, we might begin to sense a centre around which both poverty and wealth rotate; we might come to recognise these opposites as distinct faces of the industrial revolution and the mechanistic thinking that underpins it.
With machines (technology, automatic process) as the ground of our systems, the reigning values are: mass production, identical forms, standard measurements. And the requirement is: unlimited consumption, both of raw materials and finished products. An unintentional result is that the system tends to overlook and undervalue a whole set of powerful variables and realities – those which are distinctly human.
What would happen if we allowed development thinking to cut itself loose from the tentacles of a mechanised view of the world? What if, for example, we introduced a quality like humility – the kind of humility Galileo experienced when gazing at the skies – into our thinking? What if we concentrated as much on what we don’t know as on what we do know? What if we threw out the molds and conveyor belts of development initiatives and decided to let it be an exclusively human endeavour?
I think if we had the courage required for such an undertaking, three exciting things would happen:
1. We would shift the order of the “five Ws”: what, where and when would play second fiddle to WHO and WHY. To stop assuming that these last two questions always have the same generic answers (“poor people” and “we want to help”) can be powerful. These assumptions become labels and labels become reductionist. Making them more specific opens wide two windows of possibility we often forget are there. A specific and thus enriched “who” automatically introduces collaboration, partnership and diversity into our work. A deeply specific, clearly articulated “why” has the potential to keep it honest.
2. We would shift our thinking about time: when time becomes human, we notice two things. First, we get to be in charge of it. Paying attention to detail, allowing time for deep thinking while planning, or shifting time frames to accommodate reality during a project, all give us sovereignty over time. Secondly, we would notice that creating wealth and health where poverty and illness have installed themselves takes time. Put simply, development projects that take a long view are the only ones that really work because, as in any relationship, commitment is a game changer.
3. We would see a different picture: when we allow development to be human, we begin to see and understand the human beings who have the most at stake in the process. And when we meet and know them, we look beyond the mere context of their lives to see how they live their lives. When we do this we see vast – seemingly unlimited – amounts of creativity, ingenuity, intelligence, courage, fortitude, to name just a few notable human qualities for which there are no standard measures.
These ideas aren’t theoretical. A few of the finest development initiatives I know of embody the human approach. In northwestern India, in a geographically isolated region with limited resources, an organisation called Shrujan has been quietly creating wealth and helping artisans sustain themselves while nurturing an integral part of their culture for almost fifty years. In South Africa, a project known as Umzi Wethu has come up with an original, dynamic and transferrable model for completely turning around the lives of vulnerable youth.
There are other examples like these. Valuing Voices exists to celebrate them and to encourage development professionals to look at them carefully and imitate what makes them so successful. Such work whispers of an unmoving sun, nudging us to imagine that there just may be another centre somewhere else in the sky: maybe even right in the heart of the enormous potential of what we label “poor”.
When Jindra invited me to write this blog, she mentioned that at a recent conference on development monitoring and evaluation, someone told her that Valuing Voices was an idea for which people were not yet ready. Hurray, I thought. That means she must really be on to something great.
Barbara Geary Truan holds a master’s degree in international relations from the Fletcher School. She worked for six years at the International Federation of Red Cross and Red Crescent Societies, and for over a decade as an evaluator for a philanthropical initiative that supports original working projects designed to better the world.
by valuingvoicesjin | Dec 11, 2014 | Africa, Aid effectiveness, Big data, Effective Philanthropy, Evaluation, Feedback loops, foreign aid, IATI, International aid, International non-profits, Local Participants, Milenium Development Goals, Open Data, Participants, self-sustainability, Sustainability, water
What should projects accomplish… and for whom?
An unnamed international non-profit client contacted me to evaluate their resilience project mid-stream, to gauge prospects for sustainable handover. EUREKA, I thought! After email discussions with them I drafted an evaluation process that included learning from a variety of stakeholders, ranging from Ministries, local government and the national University who were to take over the programming work about what they thought would be most sustainable once the project ended and how in the next two years the project could best foster self-sustainability by country-nationals. I projected several weeks for in-depth participatory discussions with local youth groups and sentinel communities directly affected by the food security/ climate change onslaught and who benefited from resilience activities to learn what had worked, what didn’t and who would take what self-responsibility locally going forward.
Pleased with myself, I sent off a detailed proposal. The non-profit soon answered that I hadn’t fully understood my task. In their view the main task at hand was to determine what the country needed the non-profit to keep doing, so the donor could be convinced to extend their (U.S.-based) funding. The question at hand became how could I change my evaluation to feed them back this key information for the next proposal design?
Maybe it was me, maybe it was the autumn winds, maybe it was my inability to sufficiently subsume long-term sustainability questions under shorter-term non-profit financing interests that led me to drop this. Maybe the elephant in the living room that is often unspoken is the need for some non-profits to prioritize their own organizational sustainability to ‘do good’ via donor funding rather than working for community self-sustainability.
Maybe donor/funders should share this blame, needing to push funding out, proving success at any cost to get more funding and so the cycle goes on. As a Feedback Lab feature on a Effective Philanthropy report recently stated: “Only rarely do funders ask, ‘What do the people you are trying to help actually think about what you are doing?’ Participants in the CEP study say that funders rarely provide the resources to find the answer. Nor do funders seem to care whether or not grantees are changing behavior and programs in response to how the ultimate beneficiaries respond” [1].
And how much responsibility do communities themselves hold for not balking? Why are they so often ‘price-takers’ (in economic terms) rather than ‘price-makers’? As wise Judi Aubel asked in a recent evaluation list-serve discussion “When will communities rise up to demand that the “development” resources designed to support/strengthen them be spent on programs/strategies which correspond to their concerns/priorities??”
We can help them do just that by creating good conditions for them to be heard. We can push advocates to work to ensure the incoming Sustainable Development Goals (post-MDGs) listen to what recipient nations feel are sustainable, more than funders. We can help their voices be heard via systems that enable donor/ implementers to learn from citizen feedback, such as Keystone has via their Constituent Voice practice (in January 2015 it is launching an online feedback data sharing platform called the Feedback Commons) or GlobalGiving’s new Effectiveness Dashboard (see Feedback Labs).
We can do it locally in our work in the field, shifting the focus from our expertise to theirs, from our powerfulness to theirs. In field evaluations can use Empowerment Evaluation. We can fund feedback loops pre-RFP (requests for proposals), during project design, implementation and beyond, with the right incentives tools for learning from community and local and national-level input so that country-led development begins to be actual not just a nice platitude. We can fund ValuingVoices’ self-sustainability research on what lasts after projects end. We can conserve project content and data in Open Data formats for long-term learning from country-nationals.
Most of all, we can honour our participants as experts, which is what I strive to do in my work. I’ll leave you with a story from Mali. in 1991 I was doing famine-prevention research in Koulikoro Mali where average rainfall is 100mm a year (4 inches). I accompanied women I was interviewing to a deep well which was 100m deep (300 feet). They used plastic pliable buckets and the first five drew up 90% of the bucket full. When I asked to try, they seriously gave me a bucket. I laughed, as did they when we saw that only 20% of my bucket was full. I had splashed the other 80% out on the way up. Who’s the expert?
How are we helping them get more of what they need, rather than what we are willing to give? How are we prioritizing their needs over our organizational income? How are we #ValuingVoices?
Sources:
[1] The Center for Effective Philanthropy. (2014, October 27). Closing the Citizen Feedback Loop. Retrieved December 2014, from https://web.archive.org/web/20141031130101/https://feedbacklabs.org/closing-the-citizen-feedback-loop/
[2] Better Evaluation. (n.d.). Empowerment Evaluation. Retrieved December 2014, from https://www.betterevaluation.org/plan/approach/empowerment_evaluation
[3] Sonjara. (2016). Content and Data: Intangible Assets Part V. Retrieved from http://www.sonjara.com/blog?article_id=135
by valuingvoicesjin | Nov 18, 2014 | Accountability, Africa, Credit, Ethiopia, Evaluation, ex-post evaluation, Food security, impact evaluation, International aid, LIvelihoods, Local Participants, Participation, post-project evaluation, Project design, Rural Development, self-sustainability, Sustainability, Sustainable development
What’s likely to ‘stand’ after we go? A new consideration in project design and evaluation
This spring I had the opportunity to not only evaluate a food security project but also to use the knowledge gleaned for the follow-on project design. This Ethiopian Red Cross (ERCS) project “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” (with Federation and Swedish Red Cross support) was targeted to 2,259 households in Dedba, Dergajen and Shibta through provision of crossbreed cows, ox fattening, sheep/goats, beehives and poultry which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience. ERCS had been working with the Ethiopian government to provide credit for these food security inputs to households in Tigray which were to be repaid in cash over time. During this evaluation, we met with 168 respondents (8% of total project participants).
Not only were we looking for food consumption impacts (which were very good), and income impacts (good), we also probed for self-sustainability of activities. My evaluation team and I asked 52 of these participants more in-depth questions on income and self-sustainability preferences. In Tigray, Ethiopia, we used participatory methods to learn what they felt they could most sustain themselves after they repaid the credit and the project moved on to new participants and communities.
We also asked the to rank what input provided the greatest source of income. The largest income (above 30,000 birr or $1,500) was earned from dairy and oxen fattening, while a range of dairy, oxen, shoats and beehives provided over half of our respondents (40 people) smaller amounts between 1,000-10,000 birr ($50 to $500).
![]()
And even while 87% of total loans were for ox fattening, dairy cows (and beehives) which brought in farm more income, and only 11% of loans were sheep/goats (shoats) and 2% for poultry, the self-sustainability feedback was clear. In the chart below, poultry and shoats (and to a lesser degree, ox fattening) were what men and women felt they could self-sustain. In descending order, the vast majority of participants prioritized these activities:
![]()
To learn more about how we discussed that Ethiopian participants can self-monitor, see blog.
So how can such a listening and learning approach feed program success and sustainability? We need to sit with communities to discuss the project’s objectives during design plus manage our/ our donors’ impact expectations:
1) If raising income in the short-term is the goal, the project could only have offered dairy and ox fattening to the communities as their incomes gained the most. Note, fewer took this risk as the credit for these assets was costly.
2) If they took a longer view, investing in what communities felt they could self-sustain, then poultry and sheep/goats were the activities to promote. This is because more people (especially women, who preferred poultry 15:1 and shoats 2:1 compared to men ) could afford these smaller amounts of credit as well as the feed to sustain them.
3) In order to learn about true impacts we must return post-project close to confirm the extent to which income increases continued, as well as the degree to which communities were truly able to self-sustain the activities the project enabled them to launch. How do our goals fit with the communities’?
What is important is seeing community actors, our participants as the experts. It is their lives and livelihoods, and not one of us in international development is living there except them…
What are your questions and thoughts? Have you seen such tradeoffs? We long to know…
[*NB: There were other inputs (water, health, natural resource conservation) which are separate from this discussion.]
by valuingvoicesjin | Oct 7, 2014 | Accountability, Big data, Donors, Evaluation, ex-post evaluation, impact evaluation, International aid, International non-profits, JICA, Local Participants, OECD, Participation, post-project evaluation, Results, Sustainability, Sustainable development, USAID, World Bank
Data for whose good?
Many of us work in international development because we are driven to serve, to make corners of the world better by improving the lives of those that live there. Many of us are driven by compassion to help directly through working ‘in the field’ with ‘beneficiary’/ participants, some of us manifest our desire to help through staying in our home countries, advocating to powers that be for more funding, while others create new technologies to help improve the lives of others all over the world. Some of us what to use Western funds and report back to our taxpayers that funds were well-spent, others want to create future markets via increasing globally-thriving economies. We use data all the time to prove our case.
USAID has spent millions on USAID Forward and monitoring and evaluation systems. Organizations such as 3ie rigorously document projected impact of projects while they are being implemented. Japan’s JICA and the OECD are two of the rarest kinds of organizations – returning post-project to look at the continued impact (as USAID did 30 years ago and stopped). Sadly the World Bank and USAID have only done one post-project evaluation each in the last 20 years that drew on communities’ opinions. While a handful of non-profits have used private funds to do recent ex-post evaluations, the esteemed American Evaluation Association has (shockingly) not one resource.
Do we not care about sustained impact? Or are we just not looking in the right places with the right perspective? Linda Raftree has a blog on Big Data and Resilience. She says, “instead of large organizations thinking about how they can use data from afar to ‘rescue’ or ‘help’ the poor, organizations should be working together with communities in crisis (or supporting local or nationally based intermediaries to facilitate this process) so that communities can discuss and pull meaning from the data, contextualize it and use it to help themselves….” Respect for communities’ self-determination seems to be a key missing ingredient.
As an article from the Center for Global Development cites the empowerment that data gives citizens and our own international donors knowledge by which to steer: Citizens. When statistical information is released to the public through a vigorous open government mechanism it can help citizens directly. Citizens need data both to hold their government accountable and to improve their private decision-making. (On the CGD website, see discussions of the value of public disclosure for climate policy here and for AIDS foreign assistance here.)
In my experience, most communities have information but are not perceived to have data unless they collect it using 'Western' methods. Having data to support and back information, opinions and demands can serve communities in negotiations with entities that wield more power. (See the book “Who Counts, the power of participatory statistics” on how to work with communities to create ‘data’ from participatory approaches). Even if we codify qualitative (interview) data and quantify it via surveys, this is not enough if there is no political will to make change to respond to the data and to demands being made based on the data. This is due in some part to a lack of foundational respect that communities’ views count.
Occasionally, excellent thinkers at World Bank 'get' this: "In 2000, a study by the World Bank, conducted in fifty developing countries, stated that “there are 2.8 billion poverty experts: the poor themselves. Yet the development discourse about poverty has been dominated by the perspectives and expertise of those who are not poor … The bottom poor, in all their diversity, are excluded, impotent, ignored and neglected; the bottom poor are a blind spot in development." (This came from a session description for the 2014 World Bank Spring Meetings Civil Society Forum meetings, where I presented for Valuing Voices this spring, see photo below).
And as Anju Sharma’s great blog on community empowerment says, “Why do we continue to talk merely of community “participation” in development? Why not community-driven development, or community-driven adaptation, where communities don’t just participate in activities meant to benefit them, but actually lead them?” Valuing Voices would like to add that we need participatory self-sustainability feedback data from communities documenting Global Aid Effectiveness, ‘walking’ Busan’s talk. Rather than our evaluating their effectiveness in carrying out our development objectives, goals, activities and proposed outcomes, let’s shift to manifest theirs!
Our centuries-old love affair with data is hard to break. Fine, data has to inform our actions, so let’s make it as grassroots, community-driven as possible, based on respect for the knowledge of those most affected by projects, where the rubber hits the road. While that may make massive development projects targeted at hundreds of thousands uniformly… messy… but at least projects many be more efficacious, sustainable and theirs. What do you think?
by Jindra Cekan | Jun 2, 2014 | Accountability, Bilateral organizations, Donors, Evaluation, ex-post evaluation, impact evaluation, International aid, International non-profits, Kenya, Local Participants, Open Data, post-project evaluation, Resilience, Results, Rural Development, Sustainability, Sustainable development, USAID
Sustainability SPRINGing out all over the place… and Disrupting
So what is sustainability? You may think it's the climate's long-term wellbeing and how to gauge changes to that. You may think it's linked to sustainable development regarding consumption, trade, education and environment and how to assess it. You may think it's data-driven organizational success as Chelsea Clinton describes, or is it Michael Porter's business' view of Creating Shared Value on social and environmental concerns or is it about people, as hallowed University of Cambridge trains experts in its Institute for Sustainability Leadership (I revel that I was a Fellow there in the '90s). Finally, is it WCED’s lovely definition "Sustainable development is development that meets the needs of the present without compromising the ability of future generations to meet their own needs"? Yes, when applied to communities' abilities to self-sustainably and resiliently chart their own development!
So how are we to get there? A Sustainable Brands Conference this year gets us there through being clear about their own consumption, and USAID is no different. USAID Forward is putting their money where their keyboards are (so to speak), toward more sustainable local delivery by directing a huge 30 percent of its funding to “local solutions” through procurement in coming years. This framework is to “support the ‘new model of development’ that USAID Administrator Rajiv Shah has touted, which entails a shift away from hiring U.S.-based development contractors and NGOs to implement projects, and toward channeling money through host-country governments and local organizations to build their capacity to do the work themselves and sustain programs after funding dries up. I, and others celebrate the investments this will enable local firms to make in their own capacity, in leading development!
Of course all sorts of safeguards are needed, and ideally US firms would be providing capacity development, but shouldn’t we have been doing this all along, to move toward transferring ‘development’ to the countries themselves?
Source: GAO report
Also vital to sustainable development is learning from what works and doing more of it. USAID is finally planning to incorporate more ex-post evaluations into its toolkit of evaluating sustainability! Two weeks ago, PPL/LER shared their great new policy document- “Local systems: A framework for supporting sustained development” on how they can better incorporate local systems thinking into policy as well as DIME (Design, Implementation, Monitoring and Evaluation). Industry insider DevEx tells us "even though the agency plans to use ex-post evaluations to measure whether development projects are successful or not, these evaluations will not focus on “specific contractor performance” but instead consider the “types of approaches that contribute to more sustainable outcomes…to inform USAID’s country strategies and project design." While PVO implementing partners will not [yet?] be required to do ex-post evaluations as part of their projects, having this door cracked open is excitingly opening. Notably, it is a ‘back to the future’ moment, as 30 years ago USAID led the development world in post-project evaluations, yet in the last 24 years has done none (or at least not published any) except for the Food for Peace retrospective below, as I found in our Valuing Voices research of USAID's Development Experience Clearinghouse.
There is far more to watch. In our view, the whole development industry needs to grapple with the perceived barrier that funding ends with projects (note: a trust could be set up to document post-project impact 1, 3, 5 years later and results retained, much as 3ie does now for impact evaluations) and the view that one cannot discern attributable project impact with a time-lag of several years. Yet even the General Accounting Office is asking for longitudinal data; they reviewed USAID’s document and wants to see clear measures of success at Mission and HQ level by different indicators of local institutional sustainability and impact four years on.
Why should we care? As Chelsea Clinton of the Clinton Global Initiative puts it, "you can't measure everything, but you can measure almost everything through quantitative or qualitative means, so that we know what we're disproportionately good at. And, candidly, what we're not so good at, so we can stop doing that.
Yes! Development should be about doing more of what works, sustainably, and less of what doesn’t. USAID’s Local Systems Framework found the best could also be free, as in this one Food For Peace evaluation shows:
Returning to Chelsea Clinton, I’ll conclude by stating something obvious. She "wants to see some evidence of why we're making decisions, as opposed to the anecdotes” which is what getting post-project evaluation data from our true clients, our participants, is all about. Clinton says this will transform CGI into a smart, accountable, and sustainable support system for philanthropic disrupters around the world. USAID is radical for me, today, with their Local Systems investments… my neighborhood disrupter.
Are you such a disrupter too? Who else is one whom we can celebrate together?
