HOW TO DO A PARTICIPATORY EVALUATION

For some time now, participatory evaluation practitioners have travelled various highways and byways in search of the best route to useful participatory evaluations. However, no single map, no universal set of directions or procedures has emerged to guide the travellers to that destination.

Effective participatory evaluation practitioners remain undaunted by this situation, recognizing that participatory evaluations are context-specific, rooted in the concerns, interests and problems of programme end-users. They know that the complex process of situating an evaluation in the end-users' immediate reality is what charts the route and determines the evaluation's purpose and direction. Flexibility is their watchword.

Depending on the particular context in which the participatory evaluation approach is applied, choices must be made about the degree to which end-users can realistically participate in the process. The levels will vary, as illustrated in the table on page 14. Using these levels as benchmarks for an evaluation can help to make it more participatory.

Framework
The following participatory evaluation framework can be incorporated into UNDP programming arrangements by programme staff and their collaborators, including government offices, NGOs and community members. It consists of four basic phases:

1. Pre-planning and preparation
2. Generating evaluation questions
3. Data-gathering and analysis
4. Reflection and action

Pre-planning and Preparation
"In participatory evaluation, it is important not to belabor the issue of whether or not to try participatory evaluation. Resist waiting for the perfect time. Just take a stab at it! If you wait for the perfect conditions, you may never do it. All you can do is try and by trying you will learn from failure: Think small but find a problem that may serve five people in the world. Then celebrate for having tried to make the investigation more inclusive of people's voices."

Rajesh Tandon, Participatory Evaluation Conference, University of Massachusetts, Amherst,
Massachusetts, USA, 1985

Typically, the early stage of a participatory evaluation is managed at the institutional level far from the day-to-day lives of end-users. It should lay the foundation for a participatory evaluation that is rooted in stakeholder interests.

• In order to establish stakeholder interest in conducting a participatory evaluation, garner broad-based support by soliciting end-user input and collaboration.
• Since participatory evaluation strives for transparency, openly discuss the purpose, goals and objectives and the various supporting or competing agendas of the evaluation. Continually ask the basic question: "Who wants to know what for what purpose?"
• When stakeholder input is difficult to solicit, review secondary sources, such as programme documents, to gain an understanding of the context. Refer to qualitative and quantitative baseline data and consult with stakeholders either directly or indirectly.
• Address logistical matters, such as drafting and negotiating the TORs, identifying participatory evaluation facilitators and stakeholders, and other administrative matters.

Generating Evaluation Questions
When asked what questions she intended to investigate once in the field, the participatory evaluation facilitator responded with a blank stare. "I have no recipe, no yellow brick road to follow. In participatory evaluation, I rarely know what questions will be asked; that is up to end-users. With them, we survey their situation and share our respective fears, anxieties, hopes and dreams for the participatory evaluation. Then we form questions worth asking."

J. Campos, UNIFEM Field Notes, La Paz, Bolivia

• Discuss and decide with end-users which data collection methods have a high probability of yielding data that are useful and relevant to both outsiders and insiders.
• Assess the current research skills of the persons involved in the participatory evaluation and provide training as needed.
• Determine whether or not different methods will be needed for collecting various types of data. Consider a mix of data-gathering techniques.
• Take into account the prevailing socio-cultural and political climate. Political instability or geographic distances, for instance, can have a bearing on the logistics of a participatory evaluation effort as well as on which data collection methods are feasible. Sensitivity to the socio-cultural milieu, indigenous language issues, gender issues and cultural diversity, especially regarding cultural minority groups, is key.
• Negotiate evaluation questions with stakeholders. Workshops for large groups or field visits to the end-users' workplace or home are appropriate venues for contact.
• Negotiate data collection techniques and provide training as needed. In this phase, the participatory evaluation facilitator or participatory evaluation team works shoulder-to-shoulder with key actors.

Data-gathering and Analysis
"It is better to have less perfect but more usable data and data that can more easily be shared than to have a massive amount of data that becomes the private and often confidential possession of a few expert-specialists."

(Maria-Thérése Feuerstein, 1988)

• Design appropriate venues for meeting with end-users and working with them in a participatory manner. Workshops for large groups in which a cross-section of representative end-user groups can work together are effective for dealing with evaluations that are multi-level or multi-faceted. For more face-to-face contact, field visits can satisfy the requirement of gathering data from individual sources or from small groups working as focus groups.
• Provide thorough instruction or training for participatory evaluation team members who work as data gatherers. Triangulation, a simple research tool in which evaluation members confer frequently and regularly to cross-check, verify and validate the process and data, is an effective strategy to use during the data-gathering phase.

Reflection and Action
By entering another reality it is absolutely necessary for he or she who is going to in some way perform a very difficult exercise, an almost impossible exercise, that is to 'deknowledge-ize' ourselves. This means to forget the knowledge which we had before and to begin again. But now this time inside of the new reality or cultural frame of reference. This way the people can have more power than we do. Always, the best rule is to know that we do not know that new situation as do the people who live it. It is through our disempowerment that they are empowered a little.

Paulo Freire, 1985

The final phase of a participatory evaluation is characterized by the creation of solutions to end-user problems.

• Have the participatory evaluation group begin with the problems or evaluation questions that were originally defined and articulated by end-users. The goals of this activity are:
• to validate end-user experience by using it as the basis for future action plans rather than using outsider's plans;
• to motivate end-users to find solutions and act on them rather than avoid them; and
• to promote a sense of self-determination and sustainability through feelings of empowerment.

Suggested Steps to Make Each Phase of an Evaluation Participatory

Pre-planning and Preparation

• Outline a conceptual framework based on participatory evaluation principles.
• Define parameters for the participatory evaluation (i.e., what can and cannot be achieved).
• Assess constraints and resources or enabling and inhibiting factors.
• Identify the participatory evaluation facilitator, team members and stakeholders.
• Negotiate the purpose and objectives of the participatory evaluation with the key actors.

Generating Evaluation Questions

• Facilitate participatory workshops in, or field visits to, stakeholder workplace or residence.
• Collectively identify the main focus of the evaluation.

Data-gathering and Analysis

• Provide necessary training in data-gathering methods.
• Gather data collectively.
• Analyse data collectively.

Reflection and Action

• Prioritize problems to be solved or questions to be answered.
• Coordinate resources for resolving problems identified during the evaluation.
• Take collective action.

Practical Considerations
Resident Representatives, Deputy Resident Representatives, National Programme Officers and JPOs may be confronted with numerous questions prior to funding a participatory evaluation or in determining whether or not a participatory evaluation is appropriate.

For Which Type of Project/Programme is A Participatory Evaluation Relevant?
Projects/programmes that have a clearly identified group of end-users and beneficiaries lend themselves to experimentation with this methodology.

When Should a Participatory Evaluation Be Done?
Participatory evaluations may take place during the course of a project (usually at its mid-point), towards or at the end, or a significant amount of time (e.g., two years) after a project has been completed.

Undertaking an evaluation at mid-point offers several advantages. It represents an opportunity to take stock of a project's progress to date, its achievements and any obstacles encountered. Lessons learned can be applied and corrective action can be taken if necessary. Since mid-term evaluations are forward-looking, they can provide stakeholders with the tools to take different courses of action.

Terminal evaluations, which usually look at the longer-term impact of a project or programme, tend to be done towards the end of a project whereas ex-post evaluations are undertaken at least two years after a project has been completed. Since all of the stakeholders may not be involved or associated with the project or programme after its completion, the level of their participation in ex-post evaluations may vary considerably.

How Long Does a Participatory Evaluation Take?
The duration will vary with each project, depending on its complexity and the capacity and availability of stakeholders to participate in all aspects of the evaluation. It may be more important to view the evaluation as an activity done in phases rather than in one block of time. For example, the following phases or combination of phases could be part of the evaluation: defining the parameters of the evaluation; planning; collection of data; and analysis of data and recommendations. Usually about 10 person-weeks of consultant services (facilitator(s) should be calculated for a period lasting from three to six months.

Sufficient time should be allowed for pre-planning and setting the stage in the field with the various stakeholders. The commitment of all stakeholders is critical to a participatory evaluation and adequate time must be allocated to develop relationships of trust and an understanding of what participatory processes entail. This could be done in two phases. Phase one could include setting the stage and defining, with the stakeholders, the parameters of the evaluation. The second phase could involve the analysis of findings and proposed changes.

How Much Will a Participatory Evaluation Cost?
As might be expected, evaluation costs will vary depending on the activity, e.g., the level of expertise that exists within the project (if local experts are unavailable, it will be necessary to seek external consultants), the amount of baseline data already collected, the availability of stakeholders to participate in the evaluation process. The participatory evaluation facilitation team may be composed of national and international consultants. About ten weeks of consultant services and travel should be estimated. It may be necessary to factor in more than one trip for the facilitator if the evaluation is to take place over an extended period.

In general, evaluation costs may include the following:

• consultant services for one or more participatory evaluation facilitators
• travel and per diem costs
• communication costs
• consultant services­nationals
• travel and per diem expenses of Government representatives where the need arises
• other costs related to data-gathering
• translation costs for the final report in the local language.

What Are the Role and Key Skills of a Participatory Evaluation Facilitator?
Hiring (done by UNDP) a knowledgeable participatory evaluation facilitator is critical to a successful participatory evaluation. Not all evaluators know about or are trained in participatory methods and techniques.

Without a careful screening of candidates, a participatory evaluation runs the risk of becoming conventional, with limited stakeholder participation.

Participatory evaluation facilitators usually have an academic background in the social sciences and typically are social science researchers or development practitioners. They should have field experience, experience as educators of adults or as informal trainers and have a reasonable grasp of qualitative methods, such as participatory rural appraisal (PRA) and group dynamics techniques. They must also have the capacity to:

• listen
• guide and facilitate discussions, helping the group to ask key questions
• encourage trust
• delegate tasks and responsibilities
• plan actions to help bring together the viewpoints of various stakeholders
• create an environment of sharing and reflection.

The facilitator must act as a catalyst or stimulator, managing the evaluation without being seen as directing it.

What If the Project Was Not Designed with Participation in Mind?
Ideally, participation should be part of the project design from the beginning and embody the spirit of collaboration and interaction between and among different stakeholders throughout the course of project or programme execution. However, not all projects or programmes have been designed with participation in mind. Nevertheless, it is never too late to build in greater consultation with all stakeholders through field visits and monitoring trips or during TPRs. Consulting before taking decisions is an important step towards achieving greater participation in decision-making, in implementation and in the benefits of development actions. The level and degree of stakeholder participation will depend on a number of factors:

• context of the project
• degree of willingness and commitment on the part of all stakeholders to participate in a participatory evaluation exercise
• availability of baseline data
• availability of time and resources to enable stakeholders to collect data
• any external constraints that may impede stakeholder participation.

If the evaluation process is to be meaningful, then at the very least, stakeholders should participate in defining the parameters of the evaluation, analysing the findings and proposing solutions. Their involvement in the collection and analysis of data may depend primarily on the availability of time and resources. Ideally, the evaluation report should reflect the findings, concerns and recommendations of stakeholders. Where more conventional evaluations are carried out, external evaluators should share their findings with the stakeholders to ensure that recommendations are appropriate.

What Is the Role of the Resident Representative, Deputy Resident Representative, National Programme Officer and Junior Professional Officer?
Participatory evaluations challenge conventional UNDP and technical cooperation practices. Where Resident Representatives or government ministries or institutions anticipate that an external evaluator will do everything, there may be some initial skepticism about embracing participatory methods and practices. As one of the key stakeholders in the MONEY AND MAMBAS case study remarked:

"I must confess that I was very skeptical about this participatory evaluation thing at first. But I have been surprised at how well it has worked, and I can see that my staff have benefitted. Their relations with the community will be better. We all have a better understanding of the project as well as other issues concerning water in the communities."

Resident Representatives can play a key role by being supportive of participatory evaluation processes in the face of such skepticism. Resident Representatives or Deputy Resident Representatives will also play an important role in determining whether or not resources will be allocated for a mid-term review or evaluation.

The National Programme Officer or Junior Professional Officer (JPO) plays an important role in assessing the levels of understanding and commitment that exist concerning participatory evaluation and what such an evaluation would entail in terms of stakeholder participation. Often they will be responsible for defining the initial TOR with the participation of stakeholders. Part of the job of National Programme Officer, JPO, Resident Representative and Deputy Resident Representative will be to promote a better understanding of the benefits of participatory approaches, such as capacity-building, greater ownership of results and more effective programming.

How Should the Terms of Reference (TOR) Be Prepared?
In conventional evaluations, it is customary for UNDP to detail quite extensively the parameters of the evaluation and the issues that the external evaluator will examine.

For participatory evaluations, it is critical that the TOR provide as much flexibility as possible.

Since the stakeholders play a key role in defining the parameters of the evaluation, the TOR should help to launch the evaluation process without any second guessing of the issues that the stakeholders will place on the table. Thus, consideration should be given to holding a number of workshops with the key stakeholders:

• a planning workshop where stakeholders can define the parameters of the evaluation
• a smaller workshop for data collection
• another workshop for the analysis of data and feedback.

The TOR for the evaluation should address issues such as:

• project context
• purpose of the evaluation
• proposed methodology
• preliminary identification of the stakeholders
• an evaluation strategy that outlines the various of the evaluation, such as a planning workshop, data collection, analysis of data, and a feedback session
• duration of the participatory evaluation exercise as a whole and estimates of the time required for each phase
• composition of the evaluation team and specification of the expertise required
• responsibilities of the consultants
• resources required.

The initial TOR should be prepared with the participation of key project or programme stakeholders. The specific issues that will be examined by the stakeholders, however, should be left for the first phase of the evaluation exercise once a participatory evaluation facilitator has been identified.

Possible Reactions to the Participatory Evaluation Process
It's so confusing! Some participatory evaluation efforts can appear unorganized, feel chaotic or take meandering routes in achieving their objectives. However, a participatory evaluation is an emergent process. It takes its lead from the concerns, issues and problems that appear during discussion, dialogue and interaction with participatory evaluation partners. Confusion and/or ambiguity occur early in the evaluation and must be anticipated since such an evaluation involves collaboration and negotiation to arrive at mutual agreements between and among persons who probably have never met or worked together before.

It's out of control ! Ambiguity in the participatory evaluation approach is further heightened when the outside participatory evaluation facilitator views the participatory evaluation effort as one that is "out of her/his control". In one sense, this is ideal since one of the primary goals of a participatory evaluation is to share control of the process in order to engender ownership. The participatory evaluation process readily becomes clear once outsiders become conversant with the life struggles, understandings and sensibilities of those who have been voiceless and excluded from the development process. To give others control, be "out of control"!

But is this really learning? At its most fundamental level, the participatory evaluation approach is investigative and educational. It is about systematically involving the least powerful, least visible and least assertive actors in evaluating development efforts devised on their behalf. Drawing on their experiences and views exposes the true nature of the problems as they see them and engages them in a process of creating and articulating viable solutions to those problems.

The evaluator assumes the role of facilitator, who works to promote learning moments. The approach supports end-users in their efforts to confront, analyse and find solutions. Involving end-users means patiently waiting for their questions to emerge, but the wait can reap long-lasting benefits. Tapping their knowledge and using it to determine future plans can promote sustainability. Such evaluations can be empowering for both the end-users and the evaluation facilitators.

The poignant words of a Mayan woman from Guatemala who was involved in a United Nations-sponsored participatory evaluation demonstrate the value of participation:

"I didn' t think my simple words would be important. I didn't think your big bosses would accept them. But I know that without my words you would not understand our reality. You have made me think that what I think in my head is worth the time. You have made me remember what I had forgotten to think about."