Reblog: The Name of the Game is “Sustainability” but Does the Last Player Count?

The Name of the Game is “Sustainability” but Does the Last Player Count?

by John Lowrie, Reblog from https://www.scribd.com/document/11621086/The-Name-of-the-Game-is-Sustainability-but-Does-the-Last-Player-Count-by-John-Lowrie

Today, it is obligatory to answer the question “how will your proposed activities be sustainable after the project?” Most of us dutifully play the game and repeat various sentences that describe the measures that “should” bring sustainability about. Then it is usually left at that. We all move on to the next project and donor. Seldom does anyone look back over the passing of many years to check to see if promise has turned in to reality. At best there may be an end-project evaluation which will say that planned outcomes are achieved and “likely” to be sustained. It would be a very brave evaluator to be more committed beyond that.

My question is why do we play this game? (I should add: apart from the obvious answer we need the donors’ money!) Sustainability is much more than a ruse in a game. It goes to the very essence of what kind of organisation we belong to; plus all the others with us whether staff, supporters or beneficiaries; how we originated and what are our real long- term plans?

Many NGOs are artificially created groupings of people, unlike the first ones last century that emerged as people on a mission to address specific issues that they felt strongly about. Latterly, NGOs are now lumped in to “Civil Society” which is another recent fashionable label in development jargon. This is despite many officials and even NGO workers in Cambodia having no idea that it is supposed to encompass more than NGOs, to embrace other groups such as the press, trade unions, etc.

Many NGOs have actually been started by or for charismatic individuals, tapping in to somebody else’s cause or source of funding, rather than part of a collective if not mass movement in its own right towards a common end. Sometimes, international departing NGOs promote them, as part of “exit-strategies” to demonstrate that they leave something behind to show for their good work and all the money spent. They may have even included a plan for it in their sustainability proposal write-up. In Cambodia the “gravy train” that accompanied

UNTAC, the massive international effort to bring lasting peace and democracy in the early 90s, spawned many. Some of these NGOs have been good and stood the test of time. Many have fallen by the wayside. Others have been co-opted by political interests. Few have proper accountable self-governance structures.

◄“It’s the money that matters”, but will these NGO beneficiaries, the final players, get what they need to be sustainable?

So how can they be really sustainable? What is driving them – apart that is from the obvious – money? It is true that many do good work. They often do work that local authorities should be doing. But, I ask, towards what end? For example “poverty alleviation” is not an end in itself; for it to be sustained it needs much more than the usual 1-3 year time-frames that donors favour for their projects. Yet the usual pattern is –identify your target groups, go there, pass on whatever to them, then move on (to the next ones). There are exceptions. Lutheran World Federation (LWF) expect to work for 10 or more years in their target communities before they “graduate” and begin a systematic staged withdrawal. Mostly, however, the projects end on time or soon after, the files are closed, and that is that, at least until next time.

Meanwhile NGOs try to source repeat funding or new funding, and the game rules dictate where and how they proceed. It is only by luck rather than design that there will be a true match between what they want to do and what the donor is willing to give the money for. Usually there is accommodation, most likely on the part of the NGO; indeed some big donors explicitly rule out comments, queries, and changes to their guidelines which become grails of holiness. The most desperate NGOs have to re-invent themselves; depart from their original mission, suddenly acquire new skills in often far removed fields to stay in the game.  I have seen an election-monitoring organisation become an agency to consult displaced families affected by a road-widening. I know of one local rural development NGO in one province become a pro-citizen governance campaigner in another. Adaptability and learning to diversify are good qualities, but when they cause such radical changes within an NGO, it cannot be re-assuring for sustainability.

The nature of what an NGO does and its underlying philosophy is therefore key to sustainability. Those NGOs created to work in the immediate post-conflict or disaster emergency relief periods are prone to short-term visions, and sometimes they leave legacies which handicap development such as “dependency” and even “easy-come-easy- go” attitudes when foreign money seems plentiful. Those NGOs born in the next phase, i.e., after emergency relief, the start of infrastructure reconstruction and restoring public services and the economy, can also be equally short-term in their vision. In fact they have an inherent flaw. If they succeed, they do themselves out of a job and few want that as it would mean no future jobs and income! Even the Cambodian Government, despite an addiction to foreign aid, now maintains that some have overstayed their welcome.

These NGOs tend to be “welfare” or “service-provision” providers. They are confident in their abilities “we know best” but are they committed to passing their best skills and knowledge on and to the right people to take progress forward without them?

Cambodia is not alone in that politics plays a big part. The ruling party now has a 73% majority in the National Assembly, holds 98% of the 1,629 Commune Council Chiefdoms and 70% of Councillors. As the national and commune members form the constituency for the Senate, Provincial, and District Authorities, the party is guaranteed monopoly control. The situation is not helped by the absence of a neutral civil service or public service – in fact it is mystifying that UNTAC and every international donor has not tried to cultivate one. So we are left with what we have; we are where we are. It means that sustainability will only follow if we accept that reality. We have to engage constructively with the powers-that-be; we have to find the best people we can to work with, and through them, reach out to others. If not, eventual opposition or just plan lack of good will and support will affect the final outcome. Authorities always outlast NGOs, at least in Cambodia they do!

Cambodia did not have a good start when NGOs first came on the scene. Many of the first NGOs were human rights activists needed at that time and to a certain extent now, to expose calamitous treatment meted out to victims. Unfortunately this profile, still high in perceptions, has made many in power believe that this is only or mainly what NGOs do and stand for1. The legalistic approach to human rights where abuses are reported, perpetrators identified, and “name, blame shame” attached, cannot be a development tool in the sustainability tool-kit. It must be separated out and equally important alternative human rights approaches such as “rights-based development” which arouse less hostility co-exist with similar enthusiasm and means. NGOs and civil society will never have sustainable activities while their undoubted overall positive contribution gets little or no recognition by the people that count most when it comes to change.

NGOs are agents of change, which if mishandled does lead to suspicion, so to achieve change, NGOs must be clear in their message to persuade all (or a majority) that their change is worthwhile. If a long-term, wide cross-section commitment towards that change is not implanted, it is not sustainable. Too many development initiatives in Cambodia have resulted in temporary change. The “status-quo” reverts soon afterwards. In some cases the change is only tacitly accepted “take their money”, “go along with them” but “once they have gone, we go back to how we were, OK?” This is the opposite of sustainability.

There have to be contextually appropriate solutions, which can only be country-by- country, culture-by-culture. We should not have to operate on the basis of big international donors with their “one-size-fits-all” development policies and calls for proposals that allow just one format (theirs) that automatically favours the big international NGOs. They have their professional fund-raisers, who can knock out proposals that score high marks in the assessment/evaluation boxes, without many of the authors and assessors concerned ever going near the intended target beneficiaries! Yet be under no doubt, the words they pen and peruse answer beautifully on how such beneficiaries were involved, but who ever checks?

I may not be a good team-player in the present development game, so what am I suggesting as an alternative? First of all I would like to see fewer NGOs, ones which are smaller, more self-contained, and manageable to operate country-by country, sector-by- sector, region-by-region. They need to have good links to others elsewhere for best practice to be shared, but their core mission needs to be focussed to bring about certain defined changes; with the right people and the resources they need, and in the fullness of time. Therefore 5 years is an absolute minimum and asking for “core costs” to be provided should not be regarded as a mortal sin, as it tends to be with most donors. It has to be allowed, to be seen as value for money, and though indirect, still an essential element in bringing about that change. If not, how can sustainability be served?

Ironically, it is only with core support or independent means, that NGOs can play the game, so again favouring the big players. How else does an NGO cover its costs in preparing bids to donors? For example after UK-DfiD released their worldwide Global Transparency Fund in 2008, 450 organisations applied, and just 38 succeeded. Even a recent in-country release by UNDP for environmental projects attracted no less than 67 applications, of which just 13 won out. This means that for 412 in the DfiD case and 54 in the UNDP case who had devoted considerable effort, on top of their normal work, the process ultimately proved to be a waste of time; to be disappointing and especially for local NGOs to be discouraging.

I have two (NGO) organisations in mind while writing this article. Neither conforms to the pattern I criticize. They are different and I am not the only one to think that they are better. Both have poor disabled people as their target beneficiaries.  One is in sports. The other is in poverty alleviation/new livelihoods or careers and self-advocacy. Both are now purely local NGOs, with none of the trappings or spending power of the big international players that dominate the disability sector. In fact neither has anything like a long-term future, because of funding gaps, and so a lot of their work, despite excellent results so far, cannot be said to be sustainable. If they go out of business too soon, how will their beneficiaries stay involved, if sustainability is to be realised eventually?

Local NGO “Cambodia National Volleyball League Disabled2” is the sports one. Most interestingly, it is taking a route party by choice, partly by necessity, towards the private sector or “corporate social responsibility” funds in its hope to realise sustainability. CNVLD has earned a worldwide reputation for transforming the self-esteem of disabled athletes. They actually enjoy high standing on the world stage unlike their “able-bodied” compatriots. They play volley-ball, pursue wheel-chair racing and some athletes may well qualify for the 2012 London Paralympics, even if there may be no money to support them to go and compete there.

◄Disabled Sports Athletes of NGO CNVLD playing the game at its best – but do they have a future?

Now CNVLD’s quest is to raise such money and cover their modest (compared to INGO) costs. Yet this laudable aim is treated with derision by some in the sector. Pursuing private funding is viewed as an anathema to many who see it as contradictory to the “not- for-profit” concept. Does this make sense? Does such obstruction make for sustainability? Is it a sin not to want to depend solely on the usual institutional sources of funding? One critic is interesting. It has in its mission that all its services to beneficiaries must be provided free. No charges whatsoever must be levied even for those who can afford to pay! In Cambodia, everybody pays, even when services are supposed to be free3. Even if a direct charge is not levied, recipients expect and are expected to show their gratitude4, and do so especially the poorest who see it as an inescapable obligation. Surely sustainability means (a) people who can afford to pay should do so, and (b) NGOs who can attract funding and not depend on taxpayers’ money should be welcomed5?

The second organisation, New Horizons Society6, is not as lucky as CNVLD, as they cannot go down the route of sponsorship which is an accepted feature of sport. This is an NGO that did localise from an INGO but on their own terms. They voted against joining another national body created by and favoured by the big international disability NGOs. Over 200 of their Focus Persons (Group Leaders) voted in a secret ballot to form their own NGO in order to stay true to their close-knit grassroots upwards origins and growth. Now they have 3,175 members in 135 self-help groups federated up to provincial level and going on to the national stage. They have remarkable accomplishments in creating new livelihoods for their [once] ultra-poor members and have accumulated over $130,000 in revolving funds. Individual lives have been transformed. One boy has gone from beggar in the market to national singing celebrity. One young man went from lonely at home; never seen a computer; no English, to being one of today’s high-flying geeks. His class-mate, from a similar start, went on to become the Publicity Officer for CNLVD and to lead her own troupe of dancers in the NHS Child Advocacy Group performing at international conferences.

◄NGO Child Advocates demonstrate “We can do” but will donors let them?

The 135 groups went from fear of talking to officials to successful advocacy. They started with the right to education and health-care for disabled youngsters, and once they were confident went on to persuade ministers to take action to stop their meagre pension rights being denied to them. These people chose not to ask for pity, or welfare, or for service provisions to be given to them.  Instead they ask simply for the chance to show that “they can do” and that they can be self-sufficient when given the opportunities and means.

Their problem is that they cannot do this for all members yet, let alone go on to include others in the same fate as they once were. Yet despite their accomplishments, right now they can only win project activity funds. Donors refuse to pay more than 20% for running costs. Some specify as little as 5%. That does not even cover the running costs of their multi-purpose centre where meetings, training, sports, dancing, computer classes etc., go on. It may stretch to pay modest salaries to 2 or 3 staff, but they have to depend on consultant/advisors like me to help them voluntarily. Their entire organisation is radically different from the familiar set-up to be seen in international disability, donor and development organisations. There is not an air-conditioner, land-cruiser, or voucher paying university fees of expatriate’s children, etc., in sight. Yet despite its low cost and high yield, it is not yet sustainable within the present rules of most donors. If it can get over its current shortfalls and continue to build the revolving funds in to a sizeable trust or investment fund, with many members whose incomes have grown sufficiently able to subscribe fees to run their organisation, then they may have an independent viable future. But is there a donor who can adopt such a long-term vision; who will give enough to meet the real needs they identify, and who will then stay the course even when inevitable setbacks happen on the way?

So finally sustainability should be expressed in one simple notion and by the last player. It should be a measure of the change made in individual lives and over life-times of beneficiaries and their families. They are the last players in the game – who else but them can make that assessment? The present game means that it is the other players that have most say. They arrive on the scene much earlier; play in their own compact time-frames, and to their own rules. Then as external entrants, they depart the scene as soon as they can. Sustainability?

_____________________________________________

Footnotes:

1 The perception is not helped by the limitations of the Khmer language. When the word “advocacy” was first introduced “tasumateh” was used, literally “to struggle for” associated with confrontation, not partnership, and so it was viewed by many as an alien Western concept.

2 info@standupcambodia.net or www.standupcambodia.net

3 Understanding pro-poor political change:the policy process Cambodia by Caroline Hughes and Tim Conway, Overseas Development Institute, 2003 – DfiD Publication.

4 It is this “tradition” that is at the heart of a current dispute within the UN-backed Khmer Rouge

Trials, where Cambodian staff are alleged to have paid a proportion of their salary to officials with a role in their appointment.

5 Subject of course to disclosure and transparency.

6 nhs@camintel.com or www.newhorizonsunlimited.org

 

Pineapple, Apple- what differentiates Impact from Sustainability Evaluation?

Pineapple, Apple- what differentiates Impact from Sustainability Evaluation?

There is great news.  Impact Evaluation is getting attention and being funded to do excellent research, such as by the International Initiative for Impact Evaluation (3ie), by donors such as the World Bank, USAID, UKAid, the Bill and Melinda Gates Foundation in countries around the world.  Better Evaluation tell us that “USAID, for example, uses the following definition: “Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other that the intervention that might account for the observed change.”

William Savedoff of CGD reports in Evaluation Gap reports that whole countries are setting up such evaluation institutes:  “Germany’s new independent evaluation institute for the country’s development policies, based in Bonn, is a year old.  DEval has a mandate that looks similar to Britain’s Independent Commission for Aid Impact (discussed in a previous newsletter ) because it will not only conduct its own evaluations but also help the Federal Parliament monitor the effectiveness of international assistance programs and policies. DEval’s 2013-2015 work program is ambitious and wide – ranging from specific studies of health programs in Rwanda to overviews of microfinance and studies regarding mitigation of climate change and aid for trade.” There is even a huge compendium of impact evaluation databases.

There is definitely a key place for impact evaluations in analyzing which activities are likely to have the most statistically significant (which means definitive change) impact. One such study in Papua New Guinea found SMS (mobile text) inclusion in teaching made a significant difference in student test scores compared to the non-participating ‘control group’ who did not get the SMS (texts).  Another study, the Tuungane I evaluation by a group of Columbia University scholars showed clearly that an International Rescue Committee program on community-level reconstruction did not change participant behaviors. The study was as well designed as an RCT can be, and its conclusions are very convincing.  But as the authors note, we don’t actually know why the intervention failed. To find that out, we need the kind of thick descriptive qualitative data that only a mixed methods study can provide.

Economist Kremer from Harvard says ““The vast majority of development projects are  not subject to any evaluation of this type, but I’d argue the number should at least be greater than it is now.” Impact evaluations use ‘randomized control trials’, comparing the group that got project assistance to a similar group that didn’t to gauge the change. A recent article that talks about treating poverty as a science experiment says “nongovernmental organizations and governments have been slow to adopt the idea of testing programs to help the poor in this way. But proponents of randomization—“randomistas,” as they’re sometimes called—argue that many programs meant to help the poor are being implemented without sufficient evidence that they’re helping, or even not hurting.”  However we get there, we want to know – the real (or at least likely)- impact of our programming, helping us focus funds wisely.

Data gleaned from impact evaluations is excellent information to have before design and during implementation.  While impact evaluations are a thorough addition to the evaluation field, experts recommend they be done from the beginning of implementation. While they ask “Are impacts likely to be sustainable?”, and “to what extent did the impacts match the needs of the intended beneficiaries?” and importantly “did participants/key informants believe the intervention had made a difference?” they focus only on possible sustainability, using indicators we expect to see at project end rather than tangible proof of sustainability of the activities and impacts that communities define themselves that we actually return to measure 2-10 years later.

PineappleApple

That is the role for something that has rarely been used in 30 years – for post-project (ex-post) evaluations looking at:

  1. The resilience of expected impacts of the project 2, 5, 10 years after close-out
  2. The communities’ and NGOs’ ability to sustain which activities themselves
  3. Positive and negative unintended impacts of the project, especially 2 years after, while still in clear living memory
  4. Kinds of activities the community and NGOs felt were successes which could not be maintained without further funding
  5. Lessons for other projects across projects on what was most resilient that communities valued enough to do themselves or NGOs valued enough to get other funding for, as well as what was not resilient.

Where is this systematically happening already? There are our catalysts ex-post evaluation organizations, drawing on communities’ wisdom. Here and there there are other glimpses of ValuingVoices, mainly to inform current programming, such as these two interesting approaches:

  • Vijayendra Rao describes how a social observatory approach to monitoring and evaluation in India’s self-help groups leads to “Learning by Doing”– drawing on material from the book Localizing Development: Does Participation Work? The examples show how groups are creating faster feedback loops with more useful information by incorporating approaches commonly used in impact evaluations. Rao writes: “The aim is to balance long-term learning with quick turnaround studies that can inform everyday decision-making.”
  • Ned Breslin, CEO of Water For People talks about “Rethinking Social Entrepreneurism: Moving from Bland Rhetoric to Impact (Assessment)”. His new water and sanitation program, Everyone Forever, does not focus on the inputs and outputs, including water provided or girls returning to school. Instead it centers instead on attaining the ideal vision of what a community would look like with improved water and sanitation, and working to achieve that goal. Instead of working on fundraising only, Breslin wants to redefine the meaning of success as a world in which everyone has access to clean water.

We need a combination. We need to know how good our programming is now through rigorous randomized control trials, and we need to ask communities and NGOs how sustainable the impacts are.  Remember,  99% of all development projects worth hundreds of millions of dollars a year are not currently evaluated for long-term sustainability by their ultimate consumers, the communities they were designed to help.  

We need an Institute of Sustainable Evaluation and a Ministry of Sustainable Development in every emerging nation, funded by donors who support national learning to shape international assistance. We need a sustainability global database, mandatory to be referred to in all future project planning. We need to care enough about the well-being of our true client to listen, learn and act.

What can we learn from Ex-Post (Post-Project) Evaluations?

 

What can we learn from Ex-Post Evaluations?

 

In trying to learn more about sustainable development solutions, the first place to look for information is in ex-post evaluations, also commonly called post-project evaluations, which are conducted by either development organizations themselves or by independent external evaluators. Unlike final project evaluations, which are completed at the time of a project’s conclusion to assess whether or not it has achieved its intended goals, an ex-post evaluation is conducted in the years after a project’s official end date – maybe one, three, or five years after the fact. An ex-post evaluation is a highly valuable tool for determining not just how successful a development project may have been after resources and international funding were withdrawn, but rather the long-term sustainability of the outcomes for the community members who were being ‘developed’.

With the seemingly obvious necessity for ex-post evaluations to gaining a better understanding of both positive and negative development practice, I was surprised by how hard it was to actually find any. Some organizations are diligent about conducting post-project evaluations and documenting the results for future reference, namely the development assistance organization Japan International Cooperation Agency (JICA), which has an extensive reference database to search its ex-post evaluations. However, this is certainly not the norm (yet), or if organizations are conducting ex-post evaluations they are not making the information widely available to the public. My research process included search terms such as, “ex-post evaluations by international development organizations”, “post-project evaluations”, and “impact evaluations.” In using these generic search terms, I was only moderately successful in finding helpful evaluations for my reserch, which suggests the need for more readily accessible information to the public about development outcomes.

We also found that some organizations had completed these evaluations, but they were at times too vague to obtain much useful information from. Out of about 10-15 evaluations that we found so far, there were only around 7 that were clear and organized enough to include in my table of summaries. (My search was limited to projects that were conducted predominantly at the community level, rather than at the municipal or state level.) The variable quality of these evaluations has a negative impact on their usefulness – if an ex-post evaluation is in an unsearchable format or doesn’t follow a fairly standardized organization, how will it be able to inform future projects efficiently? Additionally, it would be much easier for project coordinators to learn from past projects, and even other organizations, if there existed a more accessible and methodical database to make searching for ex-post evaluations simple. Despite these challenges, I have included five different evaluations from my preliminary research with which I was able to compare results for a better understanding of how to achieve sustainable project outcomes. The framework used for analyzing these evaluations considered:

  • The sector of the development project (i.e. food security, poverty reduction, agricultrual development);
  • the implementing organization and the evaluating organization (if it was different);
  • the dates and gap between the project and the ex-post evaluation;
  • the project objectives;
  • specific ex-post evaluation methods;
  • the positive/sustainable outcomes;
  • the negative/unsustainable outcomes;
  • the transfer to authorities;
  • the amount of money invested overall;
  • and the level of local participation.

The five evaluations analyzed include:

For a full summary of these evaluations, please see the Ex-Post Evaluations Summary Table. Here, are brief synopses of the most pertinent information for the above framework of analysis, and the table provides a better context for our conclusions.

Here are the key findings from the various ex-post evaluations that we found to be most significant:

  • Over 18 million USD were spent on the five combined projects, but most projects did not explicitly enumerate how many people/households were impacted by the individual projects. An exception to this is the project in Mauritius, which reported reaching around 3,500 people. Without understanding the scale of the program, it is difficult to compare projects directly to one another.
  • Mercy Corps’ MILK Project in Niger was inclusive and participatory in its ex-post evaluation process, which resulted in hard data that can easily be analyzed, compared, and learned from in future projects. In addition, this evaluation utilized a unique pictoral tool developed specifically to include all project participants in the feedback loop, despite widespread illiteracy, so that every individual had the opportunity to provide their insight on project impacts.
  • JICA’s Ethiopian agricultural development program involved community participation in the project from the earliest planning phases, with 100% of members reporting that they had “participated” or “actively participated” in the process. This resulted in feelings of greater personal ownership of the project, and heightened local understanding of their responsibilities.
  • Evaluations that included direct community feedback in their analyses were by far the most helpful when trying to determine sustainability. For instance, in JICA’s Agricultural Development Project in the Kambia District of Sierra Leone there was no mention of local level involvement throughout any of the stages of project planning, implementation, or evaluation, which could have influenced why the project only “somewhat” achieved its objectives
  • Projects that have flexible agendas, willing to change with the changing needs of the population during the planning/implementation phases, are viewed positively by the developing community and achieve more successful outcomes. This willingness to adapt was what characterized the project in GVC OLNUS Argentine Puna. Considering the true, up-to-date needs of the community allowed for greater local participation that enabled the strengthening of local autonomy (and thus, sustainability).
  • None of the project evaluations provided a breakdown of how successful budget allocation was. The JICA projects included a breakdown of the overall budget into equipment and local costs, however despite some evaluations noting who provided certain funding, none mentioned if parts of the budget were inefficiently used. We believe it would be helpful to include not just how much money was invested in a project, but also how much of that budget either prompted direct growth or failed to produce an effective outcome.

Local community members are often referred to as ‘beneficiaries’ in the development process, yet they are the ones who governments, NGOs, and multilateral organizations are trying to empower through their various socioeconomic development missions. So, when we need to understand what worked with a project, and as importantly what didn’t work for a project, it is the voices of the community that need to be heard. A lot of great work is being done in international development, but it is clear that after her initial research that ex-post evaluations are essential to determining project sustainability and that projects that propose community-level development must also take the time to directly involve those community members in their own evaluation process. This feedback loop has the power to inform and influence future projects, while also creating the opportunity to actually listen to what participants (not beneficiaries) can sustain for themselves to achieve a better life.

Where have you found feedback loops that work? What excellent programming can you share?

 

Sources:

[1] Nishimaki, R., Kunihiro, H., & Tahashi, S. (2008, July 8). Evaluation Result Summary: The Project for Irrigation Farming Improvement. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/project/term/africa/c8h0vm000001rp75-att/ethiopia_2008_01.pdf

[2] Kumagai, M., Otsuka, M., & Sakagami, J. (2009, September 26). Evaluation Result Summary: The Agricultural Development Project in Kambia in the Republic of Sierra Leone. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/project/term/africa/c8h0vm000001rp75-att/ethiopia_2008_01.pdf

[3] The Improve Group. (2012, December). Post Project Evaluation of Mercy Corps’ MILK Program in Niger: Examining Contributions to Resilience. Retrieved from https://www.yumpu.com/en/document/read/35930718/niger-milk-post-project-evaluation-final-report-mercy-corps

[4] Proatec SRL. (2013, March). Ex Post Evaluation of Projects Managed by NGOs in Argentina. Retrieved from https://www.oecd.org/derec/italy/Evalutation-of-Projects-Managed-by-NGOs-in-Argentina.pdf

[5] International Fund for Agricultural Development (IFAD). (1997, August). Small-Scale Agricultural Development Project – Ex-post Evaluation. Retrieved from https://www.ifad.org/en/web/ioe/evaluation/asset/39828071