6.3 Key elements of the joint evaluation process
Generally, the suggested steps in planning and conducting a joint evaluation are the same as for any other well managed evaluation. However, there are a number of issues specific to joint evaluations that warrant greater attention.
Deciding on a joint evaluation—Is there a need?
It is important to assess whether the programme or project warrants a joint evaluation. To do so, ask the following questions:
- Is the focus of the programme on an outcome that reaches across sectors and agencies?
- Is the programme co-financed by multiple partners?
- Is the topic a contentious issue, thus calling for a balanced approach?
In addition, a discussion surrounding the purpose of the programme evaluation may be necessary. For instance, if the programme evaluation is solely for accountability purposes, it may not warrant a full-blown joint evaluation. Time constraints for the production of the evaluation report can also be an issue. Joint evaluations tend to be lengthier in process and require greater coordination efforts. Other advantages and disadvantages should be discussed both internally and with stakeholders (see Box 29 for benefits and challenges of joint evaluations).
Determining the partners—Who is key?
Like other evaluations, joint evaluations rely on national ownership and should contribute to the development of the capacity of stakeholders whenever possible. They also enable the voice of all stakeholders to be heard and help partners work together to assess the contributions of a programme or project. However, it is essential to determine the partners at an early stage to ensure their involvement and ownership. The partners could be determined by focusing on where the finances come from, who the implementing partners are, or by researching which other agencies are conducting similar work and thus may be contributing to the overall development goal or outcome. It is also important to assess the potential contributions of partners at this stage. For example, if a partner has a lot of other activities or constraints, it may not be best for them to get involved. It is always important to discuss the objectivity that partners may or may not bring to the table to ensure that the evaluation is independent and free from strong biases.
Choosing the management structure and division of labour
Effective management structures and communication systems are essential for a joint evaluation to function effectively. The following suggestions are drawn from various sources in the evaluation field:46
- Agreeing on the management structure—The recommended structure is two-tiered with a steering group that oversees the process and a smaller management group to ensure implementation goes smoothly. The steering group will normally comprise a representative from each partner organization and government entity. The steering group will meet at specific times to approve the ToR and the evaluation team, ensure oversight of the evaluation, introduce balance in the final evaluation judgements, and take responsibility for the use of results. Depending on the scope of the joint evaluation, a management group composed of technical representatives from concerned organizations or government entities should be created. The management group generally appoints one agency or an individual as the evaluation manager to handle the task of recruiting and managing the evaluation team. It is up to the commissioners of the evaluation to determine what works best within their particular context.
Box 36. Example of a management arrangement for a joint UNDAF evaluation
In a typical UNDAF evaluation, heads of agencies and key government officials may participate in the steering group, which provides overall guidance and direction to the process. M&E officers and technical officers in the management group are responsible for jointly drafting the ToR, managing the selection of evaluators and interacting with the evaluators on a regular basis. The Resident Coordinator’s office or another UN organization may be appointed as an evaluation manager, who is responsible for the day-to-day management of the evaluation process and coordination amongst participating agencies.
- Agreeing on the division of labour within the management group—The senior management of the UNDP programme unit should agree on the decision-making arrangements and the division of labour with other partners at the onset of the evaluation process. This involves determining who among the management group will take the lead role in each of the subsequent steps in the evaluation. A conflict resolution process should be determined to deal with any problems that may arise.
Agreeing on the scope of work
In general, it is more effective for all of the partners in a joint evaluation to discuss and agree upon the scope of the evaluation. Practical issues that should be clarified include the scope of the evaluation, the issues to be covered and the time-frame of the exercise. However, this is not always possible given the range of motivations for undertaking an evaluation, such as identifying lessons learned, establishing an empirical basis for substantive reorientation or funding revision, satisfying political constituencies in donor countries, or fulfilling institutional requirements that are particular to large projects.
Drafting the ToR—It is generally practical for one party to take the lead in drafting the ToR, which defines the scope of work. After a draft is produced, it should be discussed and agreed upon by the partner organizations. It is important to satisfy the interests of all parties concerned in the ToR to the extent possible. Consideration should be given to creating a common agenda reflecting priorities that balance ownership with what is feasible.
Determining whose procedures will be used—Since different organizations take different approaches to evaluation, it is important to allow flexibility to adapt and additional time to accommodate delays due to such differences. There are two common approaches to managing this issue: to agree that the evaluation will be managed using the systems and procedures from one agency, or to split the evaluation into components and agree whose systems will be used to manage which components.
Selecting the funding modality
A number of funding modalities are available for joint evaluations. If UNDP is taking the lead, the preferred approach should be for partners’ financial support to be pooled into a fund (akin to a trust fund) that is administered by one agency and that covers all costs related to the exercise. The second option, where individual partner(s) finance certain components of the evaluation while UNDP covers others (akin to parallel financing), is less preferable, as it increases transaction and coordination costs.
Box 37. Negotiating funding modalities for joint evaluations
These funding modalities mentioned earlier must be negotiated at the time of project or programme formulation with governments, and necessary resources should be ear-marked for a joint evaluation in the budget. If other donors are providing financial contributions, such discussions should take place while developing a project or negotiating the cost-sharing agreement. In order to facilitate discussions between UNDP programme units and donors, UNDP has prepared a clause for evaluation
in the standard third-party cost-sharing agreements.47
Selecting the evaluators
There are several ways to approach the selection of experts for a joint evaluation. One option is to task one of the partners with recruiting the evaluation team, in consultation with the other partners. Another option is for each of the partners to contribute their own experts. In some cases, the approach taken to the selection of experts may need to correspond to the funding modality. For example, if parallel financing is used, each partner might need to bring its own expert to the team. In cases where each party brings its own evaluators to the team, evaluators may have difficulty in reporting to one actor while serving as a member of a joint team. To resolve this issue, all of the institutions involved should agree on the identity of the team leader at the onset, or delegate a particular agency to recruit the team leader and make clear to evaluators that the independence of the team will be respected and expected.
Agreeing on the report and dissemination strategies
Different organizations follow different practices over who has the final say on what is included in the report. For a joint evaluation, it is generally easiest if partners agree that: they have the opportunity to correct factual errors in the report; where it is impossible to resolve differences on the findings and conclusions, dissenting views should be included in the report; and the conclusions and recommendations should be the responsibility of the evaluators. However, sometimes measures, such as allowing for separate evaluation products, may be beneficial for the partners who have certain accountability or reporting requirements.
Management response, follow-up and implementing recommendations
All managers must follow up on the findings and recommendations of each evaluation report in UNDP. However, this can be particularly challenging for joint evaluations, given that the internalization of the findings and implementation of the recommendations need to be done at the level of individual institutions and at the level of the partnership between them. Therefore, partners need to agree on what to do individually and collectively, and decide upon a follow-up mechanism that monitors the status of the changes being implemented. In line with the evaluation policy requirement, UNDP may select recommendations that are pertinent to UNDP and prepare a management response focusing on these recommendations.