5.3. Types of evaluation in UNDP


Independent and decentralized evaluations

UNDP support and services consist of programmes, projects, partnerships and ‘soft assistance’ such as advocacy, policy advice and coordination support, which may or may not be delivered within a project framework. Programmes and projects have results frameworks that detail the results map and intended results at the output and outcome levels. Evaluations in UNDP are carried out to adequately cover this wide range of UNDP initiatives in order to assess their worth and merit and support the organization’s learning efforts and accountability. The architecture of evaluation in UNDP, therefore, corresponds to the UNDP programmatic structure and its components.

There are two categories of evaluations in UNDP: independent and decentralized evaluations. The UNDP Evaluation Office is mandated by the Executive Board to carry out independent evaluations. They are referred to as independent since the Evaluation Office is independent from programme management and is not part of subsequent decision-making processes regarding the subject of an evaluation. The Evaluation Office is also required to conduct country programme evaluations (known as Assessments of Development Results or ADRs), regional and global programme evaluations, and thematic evaluations in accordance with the programme of work that is approved by the Executive Board.

The programme units carry out various types of decentralized evaluations and ensure that they provide adequate information about the overall performance of UNDP support in a given context. In doing so, the programme units draw from a range of evaluation types that are based on business units of their development assistance at the country, regional or global levels. These include: UNDAF; country, regional or global programmes; outcomes; thematic areas; and projects. The most common decentralized evaluations are project and outcome evaluations. The programme units do not conduct these evaluations themselves, but rather commission external evaluation consultants to do so. 

Together, these two categories of evaluations are intended to provide comprehensive information about UNDP performance at the project, programme, corporate and UN system levels, with a view to supporting sound management of UNDP initiatives and strategic direction.

Relationship between independent and decentralized evaluations

Although the institutional arrangements—including mandates, lines of accountability and operational modalities—of independent and decentralized evaluations are different, they complement and reinforce each other. For example, decentralized evaluations, particularly outcome evaluations, carried out in a given country provide a substantive basis for an independent evaluation of the country programme or the ADRs that are conducted by the Evaluation Office. Therefore, outcome evaluations and their associated project evaluations should be completed before the ADRs. Moreover, in conducting country case studies of a thematic or regional programme evaluation, the Evaluation Office may apply a meta-evaluation approach34 and draw extensively from country or region-specific decentralized evaluations. In the absence of adequate and credible decentralized evaluations, independent evaluations may have a limited evaluative basis and may require more time to collect necessary data. Similarly, evaluators for decentralized evaluations may use the analysis provided in the relevant independent evaluations and case studies as a building block for their analysis. Table 21 documents the main types of evaluations carried out in UNDP, including responsible parties mandated for carrying them out and main users of these evaluations.

Table 21. Examples of different types of evaluations carried out by the Evaluation Office and programme units

 

Mandated Responsibility for Evaluation

Evaluation Type

Strategic Plan

Programme Areas (e.g. governance, South-South cooperation)

Thematic Areas or Topics
(gender, capacity building, RBM)

Programme Evaluations

Global, Regional or Country Programme Outcomes

UNDAF Outcomes

Projects

Global Programme

Regional Programme

Country Programme

Evaluations conducted by Evaluation Office *

Evaluation Office

 

Thematic evaluations

Evaluation of Global Cooperation Framework

Evaluation of Regional Cooperation Framework

Assessment of Development Results

 

 

 

Primary users

Executive Board, UNDP management

Executive Board, UNDP management, BDP

Executive Board, UNDP management, regional bureaus

Executive Board, UNDP management, country office, national partners

 

 

 

Evaluations Commissioned by Programme Units**

BDP

Cross- programme area evaluations

 

 

 

Outcome or outcome-oriented thematic evaluations

 

 

 

Outcome or outcome-oriented thematic
evaluations

Mid-term evaluation

 

 

 

 

 

Outcome or outcome-oriented evaluations
(see section 5.3.2)

 

 

 

Project evaluations

BCPR

 

 

 

 

 

Regional bureaux

 

 

Mid-term evaluation

 

 

Other units ***

 

 

 

 

 

Country offices

 

 

 

Mid-term or end of cycle evaluation

 

Project evaluations
(e.g.  Global Environment Facility  terminal evaluations)

UNCT

 

 

 

 

 

 

 

UNDAF evaluation

 

Primary users

Management and partners

 

 

 

Management and partners

 

Management partners, donors

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

*The Evaluation Office is required to conduct all evaluations outlined in the programme of work approved the Executive Board.
**Programme units are required to conduct all evaluations planned in their evaluation plan.
*** Other units with programmatic responsibilities such as the Bureau of Management and the Partnership Bureau, as relevant.

Outcome evaluation

Outcome evaluations in UNDP assess UNDP contributions towards the progress made on outcome achievements. These outcomes are generally identified in the programme or project results frameworks to which UNDP initiatives contribute.

Outcome evaluations are undertaken to:

  • Provide evidence to support accountability of programmes and for UNDP to use in its accountability requirements to its investors
  • Provide evidence of the UNDP contribution to outcomes
  • Guide performance improvement within the current global, regional and country programmes by identifying current areas of strengths, weaknesses and gaps, especially in regard to:
    • The appropriateness of the UNDP partnership strategy
    • Impediments to the outcome
    • Mid-course adjustments
    • Lessons learned for the next programming cycle
  • Inform higher level evaluations, such as ADRs and evaluations of regional and global programmes, and subsequent planning
  • Support learning across UNDP about outcome evaluation

Outcome evaluations are strategic, addressing: broad-based linkages with development; partnerships across agencies; analysis of the external local, regional and global environment in the analysis of success; and the comparative value of UNDP and significance in development. Another distinct characteristic of outcome evaluations is that they explicitly recognize the role of partners in the attainment of those outcomes and provide critical information for the purpose of enhancing development effectiveness and assisting decision and policy making beyond a particular project or initiative. Outcome evaluations also provide a substantive basis for higher level evaluations (e.g. UNDAF evaluations) and independent evaluations conducted by the Evaluation Office. Therefore, the conduct of outcome evaluations during the programme cycle is mandatory for all programme units in UNDP.

As UNDP works in a wide range of development contexts and situations, the requirements for outcome evaluations can be fulfilled through different arrangements. For instance, joint evaluations, focusing on themes, large projects or geographical areas that address specific outcome(s) as pre-defined in planning documents (such as country, regional and global programme documents) may be considered as fulfilling requirements for outcome evaluations.

Whatever the arrangements may be, in order to meet the requirements for outcome evaluations, the evaluation must be outcome oriented. Outcome evaluations must meet the objectives to assess the following:

  • Progress towards achieving the outcome, including unintended effects of activities related to this outcome
  • The contributing factors to the outcomes
  • The contribution the UNDP has made to the outcomes
  • The effectiveness of the partnership strategy in achieving the outcomes

In consultation with relevant partners, UNDP programme units may decide which outcomes to choose and what modality to use in evaluation. The existing partnerships on the ground, the nature of the programme, planned evaluations by partners and government (so as to seek opportunities for joint evaluations) and other programme-specific factors may influence such decisions. For more details, please refer to the compendium on outcome evaluations.

Project evaluation

UNDP programme units may commission evaluations of their respective projects as needed. Managing for results requires, as a starting point, a good knowledge of projects, their effectiveness, internal and external factors affecting effectiveness, their added value and their contribution to higher level outcomes. A project evaluation assesses the performance of a project in achieving its intended results. It yields useful information on project implementation arrangements and the achievement of outputs. It is at this level that direct cause and attribution can be addressed given the close causal linkage between the initiatives and the outputs.

The primary purpose of a project evaluation is to make improvements, to continue or upscale an initiative, to assess replicability in other settings, or to consider alternatives. Therefore, although project evaluations are mandatory only when required by partnership protocols, programme units are strongly recommended to commission evaluations, particularly of pilot programmes, before replication or upscaling, projects that are going into a next phase, and projects more than five years in duration. Increasingly, project evaluations play an important role in accountability to donors and governments involved in financing projects.  For their own accountability reasons, donor agencies and other cost-sharing partners35 may request UNDP to include evaluation requirements in the UNDP-donor partnership agreements. Mid-term and final evaluations of Global Environment Facility projects are examples of project evaluations, as they are carried out within the clearly defined scope of a single project.36

When a project is undertaken in partnership with other development actors, the evaluation needs to take into consideration the objectives, inputs and contributions by each partner. The overall evaluation conclusions need to highlight how these different elements integrate to achieve the intended outputs, and what can be learned from the added value of the collaboration. Therefore, it is of central importance that UNDP and the partners involved in a project work together, voice their expectations and issues, and own the evaluation from the planning phase throughout the whole process.

Project versus outcome evaluations

There are several important differences between project evaluations and outcome evaluations, as illustrated in Table 22.

Table 22. Differences between project evaluation and outcome evaluation

 

Project Evaluation

Outcome Evaluation

Focus

Generally speaking, inputs, activities and outputs (if and how project outputs were delivered within a sector or geographic area and if direct results occurred and can be attributed to the project). (See note.) Outcomes (whether, why and how the outcome has been achieved, and the contribution of UNDP to a change in a given development situation).

Scope

Specific to project objectives, inputs, outputs and activities.
Also considers relevance and continued linkage with outcome.

Broad, encompassing outcomes and the extent to which programmes, project, soft assistance, partners’ initiatives and synergies among partners contributed to its achievement.

Purpose

Project based, to improve implementation, to re-direct future projects in the same area or to allow for upscaling of project.

To enhance development effectiveness, to assist decision-making, to assist policy making, to re-direct future UNDP assistance, to systematize innovative approaches to sustainable human development.

Source: UNDP, Guidelines for Evaluators, 2002

Note: Large projects may have outcomes that can be evaluated.  Further, small projects may also make tangible contributions to the achievement of CPD outcomes or even project-specific outcomes. In such instances, these project evaluations may be considered to be fulfilling requirements for outcome evaluations.

The increasing focus on outcome evaluations in UNDP does not mean that outcome evaluations have replaced project evaluations. Many programme units continue to undertake project evaluations because they yield useful information on project implementation arrangements, administrative structures and the achievement of outputs. Further, project evaluation provides a basis for the evaluation of outcomes and programmes, as well as for programme and thematic evaluations conducted by the Evaluation Office, and for distilling lessons from experience for learning and sharing knowledge.

To ensure the relevance and effective use of evaluation information, evaluations should be made available in a timely manner so that decision makers can make decisions informed by evaluative evidence.

Thematic evaluations 

In addition to project and outcome evaluations, senior managers of programme units may choose to commission thematic evaluations to assess UNDP performance in areas that are critical to ensuring sustained contribution to development results. They may focus on one or several cross-cutting themes that have significance beyond a particular project or initiative. Examples of thematic evaluations commissioned by programme units include the evaluation of UNDP initiatives in a particular results area, such as democratic governance, and the evaluation of a cross-cutting theme, such as capacity development or gender mainstreaming in UNDP programming in a given country.

Country, regional or global programme evaluation

Country offices may commission a country programme evaluation to assess UNDP attainment of intended results and contributions to national development results in a given country. The evaluation examines key issues that are similar to those in the ADRs, such as UNDP effectiveness in delivering and influencing the achievement of development results and UNDP strategic positioning. The country programme evaluation contributes to the greater accountability of UNDP and the quality assurance of UNDP initiatives at the country level. As in the ADR, it allows findings and recommendations to feed into the preparation of subsequent programmes. It can be used to facilitate dialogue with the government and other national partners and may also provide lessons that are useful for the government in its aid management work and its relationship with other development partners. Despite a number of similarities, country programme evaluations commissioned by country offices are distinct from the ADRs in terms of their scope and management arrangements. They are usually focused on a given programme cycle with a greater focus on performance at the project level. Further, decentralized country programme evaluations are commissioned by those responsible for programme management, as opposed to the independent Evaluation Office.

Similarly, regional bureaux and policy and practice units may decide to carry out mid-term evaluations of their respective regional and global programmes. These mid-term programme evaluations allow for mid-course adjustment of programmes and also feed into the regional and global programme evaluations that the Evaluation Office is mandated to conduct towards the end of the programme period.

Impact evaluation

An impact evaluation is an evaluation of the effects—positive or negative, intended or not—on individual households and institutions, and the environment caused by a given development activity such as a programme or project. Such an evaluation refers to the final (long-term) impact as well as to the (medium-term) effects at the outcome level.

By identifying if development assistance is working or not, impact evaluation also serves the accountability function. Hence, impact evaluation is aligned with RBM and monitoring the contribution of development assistance toward meeting the MDGs. An impact evaluation is useful when:

  • The project or programme is functioning long enough to have visible effects
  • The project or programme has a scale that justifies a more thorough evaluation

Impact evaluation does not simply measure whether objectives have been achieved or assess direct effects on intended beneficiaries. It includes the full range of impacts at all levels of the results chain, including ripple effects on families, households and communities, on institutional, technical or social systems, and on the environment. In terms of a simple logic model, there can be multiple intermediate (short and medium term) outcomes over time that eventually lead to impact—some or all of which may be included in an evaluation of impact at a specific moment in time. 

This definition emphasizes the need for understanding the consequences of development initiatives in the longer term.  Another important issue connected to impact evaluation is attribution—that is, determining to what extent an initiative, rather than other external factors, has contributed to observed impacts. There are many methods that can be applied to deal with the attribution issue. It is important that this issue be taken into account in the design of the initiative, as well as the evaluation ToR and design.37

UNDAF evaluation 38

UNDP programmes, projects and operations operate in concert to support UNDAF objectives and outcomes that address national priorities. The UNDAF describes the collective response of all UN operations in a country. While UNDP outcome evaluations focus on CPD outcomes, UNDAF evaluations focus on UNDAF outcomes, their contributions to national priorities and the coherence of UNCT support. The UNDAF evaluation is timed to provide inputs to the preparation of the next UNDAF, country programmes and projects by individual agencies. The UNDAF evaluation should take place at the beginning of the penultimate year of the programme cycle and build on UNDAF annual reviews as well as major studies and evaluations that have been completed by individual agencies.  Although the results of the UNDAF evaluation are meant to contribute to managing for results, it is an external function, which should be separated from programme management.  UNDAF monitoring and evaluation should always be aligned with existing national monitoring and evaluation systems or focus on their development and institutionalization if they are premature or absent.

The scope of the UNDAF evaluation depends on the previous evaluations and studies already conducted during the cycle and on the nature of UNCT operations in a country. UNDAF evaluations are jointly commissioned and managed by the heads of UN organizations and national governments. They are conducted by external consultants selected by mutual agreement between the United Nations and the government through a transparent and thorough selection process. The 2007 CCA and UNDAF Guidelines39 should be consulted for more information.

Box 28. Categorizing evaluations by timing
Evaluations can be defined in terms of different modalities of UNDP support, such as project, programme, and also different levels or frameworks of results such as outcome, UNDAF and themes. Evaluations can also be defined by when they are carried out:

  • Ex-ante evaluation is a forward-looking assessment of the likely future effects of new initiatives and support such as policies, programmes and strategies. It takes place prior to the implementation of an initiative.
  • Mid-term evaluation generally has a formative nature as it is undertaken around the middle period of implementation of the initiative. Formative evaluation intends to improve performance, most often conducted during the implementation phase of projects or programmes.
  • Final or terminal evaluations normally serve the purpose of a summative evaluation since they are undertaken towards the end of the implementation phase of projects or programmers. Summative evaluationis conducted at the end of an initiative (or a phase of that initiative) to determine the extent to which anticipated outcomes were produced. It is intended to provide information about the worth of the programme.
  • Ex-post evaluationis a type of summative evaluation of an initiative after it has been completed; usually conducted two years or more after completion. Its purpose is to study how well the initiative (programme or project) served its aims, to assess sustainability of results and impacts and to draw conclusions for similar initiatives in the future.

Evaluations defined by the initiative modality of development initiatives or level of results can be further defined by the timing. For example, a programme unit may undertake a final project evaluation or a mid-term UNDAF evaluation.

 
Real time evaluations

Real time evaluations are often undertaken at an early stage of an initiative to provide managers with timely feedback in order to make an immediate difference to the initiative. They are commonly applied in humanitarian or post-conflict contexts to provide implementing staff with the opportunity to analyse whether the initial response or recovery is appropriate in terms of desired results and process. They can also be used in crisis settings where there may be constraints in conducting lengthier evaluations. These constraints include the absence of baseline data, limited data collection efforts due to a rapid turnover of staff members (for example, lack of institutional memory) and difficulty conducting interviews and surveys due to security issues.

Joint evaluation

Joint evaluation is one modality of carrying out an evaluation to which different partners contribute. Any evaluation can be conducted as a joint evaluation.  Increasingly, UNDP is engaged in joint evaluations and there are various degrees of ‘jointness’ depending on the extent to which individual partners cooperate in the evaluation process, merge their evaluation resources and combine their evaluation reporting.40 

The joint evaluation approach became popular in the 1990s with the promotion of the approach through the DAC Principles for Evaluation of Development Assistance, which stated, “Joint donor evaluation should be promoted in order to improve understanding of each others’ procedures and approaches and to reduce the administrative burden on the recipient.”41 The Paris Declaration also reinforced the joint evaluation approach through the commitment made by development agencies and partner countries to find more effective ways of working together.42 Joint evaluations can be characterized by a number of benefits and challenges as shown in Box 29.

Box 29. Benefits and challenges of joint evaluations

Benefits

  • Strengthened evaluation harmonization and capacity development: shared good practice, innovations and improved programming
  • Reduced transaction costs and management burden (mainly for the partner country)
  • Improved donor coordination and alignment: increase donor understanding of government strategies, priorities and procedures
  • Objectivity and legitimacy: enables greater diversity of perspectives and a consensus must be reached
  • Broader scope: able to tackle more complex and wider reaching subject areas
  • Enhanced ownership: greater participation
  • Greater learning: by providing opportunities for bringing together wider stakeholders, learning from evaluation becomes broader than simply for organizational learning and also encompasses advancement of knowledge in development.

Challenges

  • More difficult subjects to evaluate (complex, many partners, etc.)
  • Processes for coordinating large number of participants may make it difficult to reach consensus
  • Lower-level of commitment by some participants
Adopted from OECD, ‘DAC Guidance for Managing Joint Evaluations’, Paris, France, 2006; and Feinstein O and G Ingram, ‘Lessons Learned from World Bank experiences in Joint Evaluation’, OECD, Paris, France, 2003.

At the country level, one of the most obvious examples of a joint evaluation is the UNDAF evaluation, in which a number of UN organizations and the government participate. In addition, a UNDP country office may jointly carry out, together with the partner government or with a donor, a joint outcome evaluation that looks where both parties are mutually and equally responsible for the evaluation exercise. For guidance on how to organize and manage a joint evaluation process, see Chapter 6.