TRAINING MODULE
INTRODUCTION
The case study described in this training module is based on UNDP's effort to conduct a participatory evaluation in an actual country setting. Participatory evaluation, an innovative approach to evaluation, can contribute to sustainable development by involving key stakeholders in assessing programmes and projects from their perspective. Involving stakeholders at all levels of evaluation can lead to more comprehensive assessments of development and can effectively draw upon beneficiary views and opinions to redirect development planning.
Purpose
The module is designed to:
Audience
This training module targets UNDP personnel, especially Assistant Resident Representatives, junior-level personnel and programme officers. Other government or non-governmental groups or agencies involved in UNDP programme and project planning, implementation, and monitoring and evaluation are also included.
Format
The case study format used in MONEY AND MAMBAS actively involves trainees in discussions and analysis of participatory evaluation theory and practice. Working through the case study helps trainees to understand events that supported and worked against the participatory evaluation exercise. All factual details and names of individuals involved in the case study have been changed to avoid distraction from the learning objectives of the training module.
Pre-workshop Activities
Workshop Activities
The module, which has been designed to help groups to analyse the pros and cons of participatory evaluation during a single training session, is organized for a three-to-four-hour time frame but can be adapted to meet specific needs. Below is a suggested schedule with approximate times for each activity.
Introduction (30 - 45 minutes):
Small groups (30 - 45 minutes):
Break (12 minutes):
Plenary (45 minutes):
Small groups (30 - 45 minutes):
Plenary (30 - 45 minutes):
Money and Mambas1 or Listening to the People
A case study of a participatory evaluation of a water and sanitation project
This case study is based on an actual participatory evaluation effort by UNDP in Eland. It is presented in three main parts:
PART 1.
PREPLANNING AND PREPARATION
In the late 1980s, UNDP decided to conduct a number of evaluations with greater participation by local community and stakeholder groups. After an extensive review of possible projects for evaluation, the former UNDP Central Evaluation Officenow reconfigured as the Office of Evaluation and Strategic Planning (OESP)settled on a water and sanitation project in Eland.
The project chosen was called "Rural Water Supply and Sanitation". Its purpose was to supply water to homesteads in rural areas, to provide training in health and hygiene and to construct latrines.
The project appeared to have the right characteristics: during its design, there had been an effort to involve the people and when it was submitted to the programme approval committee in New York, the presenters had emphasized that it was aimed at the poor in rural areas and that it was built on community participation. Project work had started in 1990 and was scheduled for completion at the end of 1995. In addition, an evaluation was planned for mid-1994. Hence, the timing and subject appeared to be right.
In January, 1994, OESP in New York approached the Resident Representative of the UNDP office in Eland to ask if a participatory evaluation approach could be used. If so, OESP would pay the full costs of the evaluation from its funds instead of using the funds allocated on line 16 of the project budget. The Resident Representative consulted the authorities in Eland and their positive response made a participatory evaluation agreeable to all partners.
PART 2.
GETTING STARTED: GENERATING EVALUATION QUESTIONS
OESP initiates a participatory evaluation of UNDP-Eland's community-based water and sanitation project. All start-up procedures begin. Preparatory work includes a desk review by Claus and Karl of all project documents and telephone conferences between Claus and Karl and OESP in New York. Contractual agreements with the two international consultants are confirmed by OESP. Additional members of the participatory evaluation facilitation team are recruited (Lane and Didi), with the inclusion of a female sociologist (Didi) almost an afterthought.
Standard UNDP procedures require clearly defined terms of reference (TOR), including precise definition of evaluation questions and data-gathering methods. In a participatory evaluation, pre-determined evaluation questions are often considered irrelevant to the immediate needs and concerns of programme beneficiaries. Participatory evaluation asks one important, fundamental question: Whose questions are being asked in a particular evaluation? Therefore, individuals attempting a participatory evaluation are challenged starting with the TOR. In MONEY AND MAMBAS, drafting the TOR presents the first major obstacle.The TOR dilemma quickly surfaced:
Bruce North's thoughts:
One snag has already emerged. The TOR seem to be a problem. It seems that Claus and Karl (alleged participatory evaluation "specialists") insist that stakeholders determine the questions to be asked in a participatory evaluation. A novel and noble ideabut too time-consuming. I wonder about this participation fetish. Haven't we hired them to do the evaluation?
Claus and Karl in Sweden:
UNDPEland agrees with consultants that a question-generating process should be part of the TOR.
Because so much of a participatory evaluation is context-specific and cannot be pre-determined, participatory evaluation facilitators must bring a jack-of-all-trades tool kit with them when they arrive. Claus, the first consultant to arrive in Eland, is an experienced participatory evaluator but unaware of the surprises awaiting him.
March 1994. Claus's introductory meeting with the UNDP Resident Representative and programme officer was cordial yet distant. The tone of the meeting seemed non-committal and the meeting ended abruptly. His introductory meeting with a key Eland actor, Richard Kosi, was even more perplexing. Claus recorded the following thoughts:
Claus's thoughts:
My meeting with Richard revealed that no one has been invited to Workshop I, which begins in three days. Richard seems to have an exclusive invitation list which includes certain beneficiaries and excludes others. He has excluded his mid-level and field staff. What will it take to convince Richard that all stakeholders need to be represented? Four months in the planning and I feel like I'm talking to myself. I wonder if anybody in the country office or the Rural Water and Sanitation Department took the time to read the two-page briefing note on participatory evaluation and the TOR detailing the evaluation exercise that were sent ahead of my arrival. I have strong doubts.
Richard's thoughts:
Claus's thoughts:
Despite the initial confusion, Workshop 1 started promptly a few days after Claus's arrival. The usual official welcomes and introductions were led by Richard Kosi. The 24 participants in attendance included 16 women and 1 Community Water Committee leader, representing programme beneficiaries; one Ministry of Health representative; one UNDP representative; three participatory evaluation facilitation team members; and the senior engineer of the Rural Water and Sanitation Department and his deputy. The group focused quickly on the purpose of the evaluation and generated the first important list of evaluation concerns.
Examples of Problems and Questions Raised by the Stakeholders
The issues brought the participatory evaluation into sharp focus from the beneficiaries' point of view. The women's group was particularly animated and articulate. They more than anyone else grasped the participatory evaluation idea. The Rural Water and Sanitation Department's Senior Water Engineer and his staff did not participate in the lively discussions of the small groups. They did an excellent job of holding themselves aloof from the exchange of views.
Claus's thoughts:
Claus, Lane and Didi led the group through a discussion about important participatory evaluation steps, including data-gathering techniques, recording data and basic data analysis. At the day's end, Richard mentioned another problem. He announced that tomorrow's workshop session must end early to accommodate participants residing in the countryside. The rushed schedule, although a considerate logistical decision, could potentially sabotage the participatory goal of stakeholder involvement in planning the participatory evaluation.
Claus, at the end of Day 1:
Surprisingly, Day 2 progressed smoothly and quickly. The discussion in small groups generated constructive recommendations regarding data-gathering methods. Questionnaires and a work plan were developed based on questions that stakeholders had identified regarding issues to be addressed by the evaluation. The group decided to tailor questionnaires to address the concerns of particular groups and to distribute them to women homesteaders, Water Committee officers, community water-minders, and Rural Water and Sanitation Department field staff, all of whom would be responsible for gathering the data. Because of time constraints, Claus ended up having to prepare the questionnaire.
Claus to himself:
PART 3.
DATA-GATHERING, DATA ANALYSIS AND ACTION
In the interim, Lane and Didi conducted the data-gathering phase. All data were collected by Lane and Didi, who did not include other stakeholders although their inclusion had been agreed upon in Workshop 1. It seemed that villagers who represented communities in Workshop 1 and drew up the list of questions reverted to the traditional role of question-answerers, not question-makers, during the data-gathering phase instead of collecting information and asking questions that had been identified during the planning workshop.
When Karl and Claus returned to Eland to review and analyse data, they learned that only 22 water schemes had been visited, which meant that only 200 beneficiaries had been interviewed. This figure was well below what had been planned. Group interviews, and not the individual interviews that had been planned originally, had been conducted.
Karl to himself:
The four-week separation had also promoted another unexpected event. Richard unilaterally decided to reconfigure the workshop participant list. He invited more senior-level policy-makers, including Government Ministry representatives, UNDP and Rural Water and Sanitation Department officials. Principal secretaries of the Ministries of Finance and Economic Planning, Agriculture, Labour and Health were the new faces that replaced all but four of the 16 rural village women water users. In addition, a few Chiefs or their representatives had been invited.
Karl to Claus:
Despite the changes, Workshop 2 progressed well: 30 people attended, including four beneficiaries, and OESP's evaluation officer from New York attended the final session. The group collectively reviewed the data and quickly categorized the main findings into socio-cultural, technical and organizational issues. The group review process allowed workshop participants to agree with or argue the data. The presence of Chiefs representing numerous traditional community groups was particularly welcomed.
Participants sped through the final steps and organized plenary presentations, which focused on problem analysis and recommendations for future action. The following comments help to convey how some issues were resolved by the group.
Rural Water and Sanitation Department and Ministry of Health staff:
End-users:
Richard, Senior Rural Water and Sanitation Department Officer:
Clearly, end-users saw the threat and fear of mamba snakes in the pit latrine as the major problem while the Rural Water and Sanitation Department saw money and personnel as the main issues. The discussion continued:
Traditional Chiefs:
Various voices:
Rural Water and Sanitation Department field staff:
Chiefs:
Various voices:
End-users:
Chiefs:
Workshop participants spoke frankly and openly and challenged each other on important issues. The participatory evaluation facilitators had hoped that the final session would result in conclusions and recommendations for future action. This was not accomplished. Facilitators found that the exercise had generated more data and information than was expected and the final session did not allow for a thorough discussion of all issues. Yet four clear action points were identified by the group: (a) the need for user designed latrine construction; (b) improved community training for water committees; (c) improved Rural Water and Sanitation Department-community coordination; and (d) improved relations with traditional power structures.
After the workshop, plans for the facilitation team's remaining days in Eland were made. The team produced a draft of the final report and circulated it to the Rural Water and Sanitation Department and UNDP for initial comments. Commitments were made by both offices to collaborate on the translation of the report's summary and its circulation to the participants in both workshops.
Most revealing to Karl and Claus were the lessons learned by their Eland counterparts:
Richard to Karl and Claus:
Lane:
My favourite problem was the one asking that Water Committee members not drink beer before their meetings.
Didi:
The eight-month participatory evaluation process had finally ended. Long after the final days, questions lingered. Had the process been effective? What would its long-term impact be? Could the costs be justified? Most important, what would happen in the coming months in Eland?
Money and Mambas or Listening to the People
Questions for Group Work
PREPARATION ISSUES
B. Questions about Part 3: Data-Gathering Issues
PROCESS ISSUES
ACTION ISSUES
C. Additional Questions: Application Issues
Headquarters has come up with this experimental project. I've been briefed on this by Mendisa, our National Programme Officer. Our project is somewhat on track. Our goals of supplying water to rural home steads, providing health and hygiene training, and constructing latrines have been met. The Rural Water and Sanitation Department has put in pipes and filtration systems to community standpipes. Our expansion to 22,000 water users is within reach, but one disturbing trend remains: pit latrine construction (critical to sound water and sanitation) is far below target because villagers refuse to build latrines according to government specifications. Their resistance jeopardizes the project. The original project design was a package in which the provision of water supply systems linked to rural sanitation through ventilated pit latrines is conditional upon community construction of ventilated pit latrines. We had villager buy-in at the start, but particip
Our first major obstacle came with the TOR. In participatory evaluation, pre-determined questions are typically irrelevant to stakeholders; they are questions "outside" of project beneficiaries' immediate needs. Who knows better than water users what works or doesn't? TORs for participatory evaluation should include end-users as question-makers, not just question-answerers. In participatory evaluation there is always "the chicken or the egg" dilemma: whose questions are more important, ours or theirs?
I am concerned, to say the least. UNDP does not seem to really know much about the participatory evaluation nor do they seem interested. Frankly, the apparent lack of genuine buy-in expressed so far tells me they are not interested because this "was not their baby".
This participation idea seems good. The way I see it, these Claus and Karl fellows are participatory evaluation experts being paid to carry out the entire evaluation. They are responsible for all the main aspects and for the final recommendations and writing up the report. My staff participates by helping. Claus wants all stakeholders to be involved from the start. How unrealistic this man is. Surely, only trained evaluators should collect the data. Don't these fellows get paid enough to handle everything?
As if all of these setbacks are not enough, the Resident Representative informed me that neither he nor his deputy will attend Workshop I, but he will send his programme officer. UNDP's noncommittal stance and Richard's misunderstanding of participatory evaluation put the effort into question. Most disturbing are the differences Richard and I have regarding stakeholder involvement. How could Karl and I have so wrongly assumed that everyone would jump on our bandwagon? Maybe we should have spent more time sensitizing people to the value of participatory evaluation?
Given the aloofness of the Rural Water and Sanitation Department chief officer and staffers, it appears that the process is either of no interest to them or it may appear threatening. Could I have been wrong about everyone's buy-in for the participatory evaluation? Do they fear that it may probe sensitive management issues that could put them in a bad light? Why aren't water officials interacting with the actual water usersthe women?
Day 1 has finally ended. My thoughts are scattered. The first few days were full of problems, but this is to be expected. The skepticism shown by the Rural Water and Sanitation Department officials doesn't help. There is a spirit lacking here... a core something missing. Many of Kosi's staff, some of the primary project stakeholders, were absent or were assigned to field duty by him. So much for stakeholder participation. It seems like a half-hearted effort. With Karl in Sweden and Didi not expected until tomorrow, even our facilitation team worked at half strength. What on earth might we expect for the following phases? Only time will tell.
A new understanding and interest in participatory evaluation seemed to emerge today. Almost everyone seemed more comfortable with the concept and with each other. Perhaps the evaluation sketch that Lane and I enacted yesterday contributed to this new attitude. Everyone had the chance to comment on what should be evaluated. There is less skepticism and more enthusiasm in the air. Tomorrow, I leave Eland. Karl and I will return four weeks after data have been collected. It is all in their hands now.
AprilMay 1994.
Four weeks elapsed between Workshops 1 and 2.
When I first saw the data, I strongly doubted its quality: not enough water schemes had been included; many interview reports lacked valuable responses; and the group format may have compromised the reliability. It was evident that Lane and Didi did not have sufficient background in working with empirical data and, worse, Claus and I had failed to give them sufficient long-distance support. Do we go ahead anyway?
So much for participatory decision-making. The decisions that were originally made and agreed upon by the group have been changed in our absence. Is there some hidden agenda here that we are not part of? Whose participatory evaluation is this anyway? Is this what is meant by "sharing control" in participatory evaluation? Who wants to control what for what purpose? What about the women? Will it be men who speak for the women again?
Our group questioned why villagers refuse to follow our rules of building pit latrines. Our designs are excellent and have worked in other places. What's their problem?
We object! We object to that conclusion! The Ministry of Health design is no good for us! We want latrines with walls that allow for air flow and let in the light. In the Ministry of Health design, the pit is far too deep and the walls collapse because of imprecise site and soil specifications. Worst of all, the latrines are dark and wetperfect housing and nesting ground for the venomous mamba snake! Let's see you share your latrine with a mamba!
You think the mamba snake is the worst problem? Not so! The problem has nothing to do with mamba snakes but has everything to do with not being able to retain good staff. The Rural Water and Sanitation Department is not a government ministry with an official budget. The Ministry of Finance classifies us as an externally funded project dependent on donor offerings. Our economic uncertainty undermines any long-range planning and does little to boost staff morale. In short, we cannot attract or retain the best technicians, much less technicians with a sensitivity to rural villagers. Now, that is the problem.
All problems would be solved if the Rural Water and Sanitation Department would work more closely with the traditional power structure through which we exercise authority. All coordination and entry into community affairs should go through us. This is strictly a problem of what happens when you bypass traditional authority.
No, we reject that idea. We object! Yes, we do need closer coordination between Rural Water and Sanitation Department field activities and the community groups. Training would be a good place to begin, and maybe the Chiefs should be the first to be trained. That's right. The Chiefs need training!
Many Chiefs and their Councils do not understand modern development needs. Chiefs need education, too, but they are difficult to persuade to come. If they don't come when invited, they send representatives instead.
Chiefs could be encouraged to attend training. Just offer incentives for our time.
We protest, protest, protest...!
Why do Chiefs need incentives? Isn't being a chief incentive enough? Why should Chiefs be paid to take part in training that they need to do their job? Others don't get any incentives. We don't get incentives, yet we give of our limited time to attend Water Committee meetings and even these workshops.
If the Rural Water and Sanitation Department wants to reach the Chiefs and work with the community through us, it needs to learn to work through channels it does not normally work through. We would be more likely to come if we were invited to training via the King's offices, our traditional power structure. The money is not the issue.
I must confess that I was initially very skeptical about participatory evaluation. I was surprised at how well it worked and I saw that my staff benefited. I believe their interaction with the communities they serve will improve. We all have a better understanding of the project and a new appreciation for the value of soliciting stakeholder input.
What a tremendously satisfying experience! The participatory evaluation often seemed like a logistical nightmare, but it was a revelation to me. Even after 15 years working on water issues, I was struck by the articulate precision with which women end-users pinpointed critical water issues that I had overlooked.
I was amazed at how much I learned about my country. I travelled to rural areas to see women in their natural surroundings. I was most amazed to find that the subject of water provided the entry point for discussion of community problems. Despite the Rural Water and Sanitation Department's many accomplishments, the crucial problem was simple: water containers had too small an opening, which precluded cleaning and encouraged algae growth. We learned that sanitation problems need to focus on water collectors. While visiting the homesteads, I often saw that kids, the chief water collectors, had the dirtiest little hands!
A. Questions about Parts I and 2: Stakeholder Issues