What makes it Difficult to Evaluate Projects Ex-post: Lessons from WWF
The World Wildlife Fund (WWF) generously shared lessons from two ex-posts and in particular what they felt should not have been done in terms of organizational learning. They originally wanted to see whether ”the work done is sustainable and so to provide lessons for the implementing office, donor offices and the wider WWF network about designing and implementing projects to deliver outcomes that are sustained beyond WWF’s involvement.”
They used six criteria for country/ case selection:
This was in line with Valuing Voices’ (Ex-post) Evaluability Checklists.
- Organizational considerations including interest in learning by WWF
- Methodological considerations regarding data access, quality etc.
- Choice of projects, sites, timing to ensure post-closure, contribution of results isolated
WWF Case 1:
The first WWF case examined the importance of evaluating both Sustained and Emerging Impacts (those expected or those that emerged from local efforts or other entrants post-INGO-exit) and managing the ex-post evaluation process, as LWR and World Vision have done.
There were three types of interesting results (refer to Table 1).
- Methodological problems in comparability to final evaluation and impact data as well as choice of project: problems with site/ case and lack of water data comparability as well as the entry of a new NGO garnering benefits (so limited contribution/ isolation of results). WWF tried to mitigate it with good design guidance and ongoing review processes during the ex-post.
- The emergence of new data ex-post about the benefits of the local user association for conflict management and emerging new catchment management
- There were ex-post limitations intrinsic to the species and ecosystem resources on which they rely, specifically species and water. Other problems of not tracking impacts on animals, not learning from evaluations, and not having saved documents might benefit other projects’ quality.
Table 1. WWF Case 1
Some learning might be attributable to the difficulties of evaluating ecosystems or it might provide important cautionary lessons for others considering such ecosystem evaluations.
But overall, it shows that in contexts where one of these criteria are present:
1. original programme having design flaws,
2. poor collection of evidence of progress against the chosen indicators of success,
3. poor documentation retention,
Then, there is limited value to be gained from doing an ex-post evaluation.
WWF Case 2:
The excellent news is that ex-post demonstrated that all three kinds of impact indicators, namely species numbers, forest cover, and conservation area improved (note, data are confidential, thereby not cited below) and that the XX Conservation Area is still managed by the community (Table 2). Also, the team gathered information that living standards had improved, as had, impressively a country-level seed fund and active participation in managing and scientifically learning from their resource.
The second case also highlights typical ex-post methodological evaluation considerations, including choice of sites/ timing:
- not applying a standard monitoring process for the whole programme period, which caused confusion about end of project versus ex-post impact progress
- the ex-post evaluation was planned predicated on the project finishing and staff exit from the area, but there was ongoing financial support of the CSO from the central office for a further two years which meant that this was not a true ex-post evaluation as it happened before the two-year minimum timeframe
- measurement led to ‘anecdotal’ evidence and comparisons made to national statistics rather than change from project evaluations which limits ex-post learning particularly
- there were negative findings of sustainability how well the community appreciated the XXCA and insufficient economic return on investment expectations that some costs such as trash removal were not covered by anticipated tourist revenues.
It also highlights that ecosystem ex-post evaluation is complex, as wildlife number was difficult to methodologically confirm, due to progression in monitoring technologies. A valuable question that WWF shared was “how to assess [continued] benefits to people from natural resource management?”
Table 2: WWF Case 2
WWF found and shared important and appropriate lessons about what these ex-posts brought with them. Most fell into aforementioned categories of Choice of projects, sites, timing, and importantly, the value of considering Methodological considerations regarding data access, quality from project onset.
What was new to us at Valuing Voices was that Organizational considerations including interest in learning by WWF-UK and their partners which is something we have not seen before. Equally, the importance of a solid rationale for asking a country office to participate in such an evaluation, and not imposing an ex-post on a less than enthusiastic program for whom sustainability may not be a priority. This links these cases together, as given limited budgets, WWF and their partners could choose to prioritize design spending, perhaps above final evaluations, especially if local NGOs know donors will leave at project’s end, so funding to improve their internal capacity strengthening might be a larger priority. Knowing the project has an immutable closeout can affect interest at the headquarters, as it can be quite difficult to find funding to evaluate ex-post, and more pressing current work competes. Also, implementing local NGOs/ CBOs could actually know about sustainability far more than donors, hereby eliminating their interest, as an M&E officer recounted. Both organizations could have other priorities, including new projects with new staff, who had less interest in the old project. Staff turnover is an additional, important consideration in whether to do an ex-post.
Table 3: WWF Learning
Given how vital it is to evaluate the sustainability of our environment, including ecosystems and natural capital on which our participants and partners depend, WWF is going ahead on evaluating ecosystems, adding to the vital knowledge base. The Global Environmental Facility is also experimenting on remote-monitoring for the evaluation of ecosystems. USAID’s Water Bureau has also done ex-posts on Water/ Sanitation/ Hygiene.
We have a whole world of sustainability to discover!