Annexes
|
5.3. Types of evaluation in UNDP
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Mandated Responsibility for Evaluation |
Evaluation Type |
||||||||
Strategic Plan |
Programme Areas (e.g. governance, South-South cooperation) |
Thematic Areas or Topics |
Programme Evaluations |
Global, Regional or Country Programme Outcomes |
UNDAF Outcomes |
Projects |
|||
Global Programme |
Regional Programme |
Country Programme |
|||||||
Evaluations conducted by Evaluation Office * |
|||||||||
Evaluation Office
|
Thematic evaluations |
Evaluation of Global Cooperation Framework |
Evaluation of Regional Cooperation Framework |
Assessment of Development Results |
|
|
|
||
Primary users |
Executive Board, UNDP management |
Executive Board, UNDP management, BDP |
Executive Board, UNDP management, regional bureaus |
Executive Board, UNDP management, country office, national partners |
|
|
|
||
Evaluations Commissioned by Programme Units** |
|||||||||
BDP |
Cross- programme area evaluations |
Outcome or outcome-oriented thematic evaluations |
Outcome or outcome-oriented thematic |
Mid-term evaluation |
|
|
Outcome or outcome-oriented evaluations |
|
Project evaluations |
BCPR |
|
|
|
|
|
||||
Regional bureaux |
|
|
Mid-term evaluation |
|
|
||||
Other units *** |
|
|
|
|
|
||||
Country offices |
|
|
|
Mid-term or end of cycle evaluation |
|
Project evaluations |
|||
UNCT |
|
|
|
|
|
|
|
UNDAF evaluation |
|
Primary users |
Management and partners |
|
|
|
Management and partners |
|
Management partners, donors |
||
*The Evaluation Office is required to conduct all evaluations outlined in the programme of work approved the Executive Board.
**Programme units are required to conduct all evaluations planned in their evaluation plan.
*** Other units with programmatic responsibilities such as the Bureau of Management and the Partnership Bureau, as relevant.
Outcome evaluations in UNDP assess UNDP contributions towards the progress made on outcome achievements. These outcomes are generally identified in the programme or project results frameworks to which UNDP initiatives contribute.
Outcome evaluations are undertaken to:
Outcome evaluations are strategic, addressing: broad-based linkages with development; partnerships across agencies; analysis of the external local, regional and global environment in the analysis of success; and the comparative value of UNDP and significance in development. Another distinct characteristic of outcome evaluations is that they explicitly recognize the role of partners in the attainment of those outcomes and provide critical information for the purpose of enhancing development effectiveness and assisting decision and policy making beyond a particular project or initiative. Outcome evaluations also provide a substantive basis for higher level evaluations (e.g. UNDAF evaluations) and independent evaluations conducted by the Evaluation Office. Therefore, the conduct of outcome evaluations during the programme cycle is mandatory for all programme units in UNDP.
As UNDP works in a wide range of development contexts and situations, the requirements for outcome evaluations can be fulfilled through different arrangements. For instance, joint evaluations, focusing on themes, large projects or geographical areas that address specific outcome(s) as pre-defined in planning documents (such as country, regional and global programme documents) may be considered as fulfilling requirements for outcome evaluations.
Whatever the arrangements may be, in order to meet the requirements for outcome evaluations, the evaluation must be outcome oriented. Outcome evaluations must meet the objectives to assess the following:
In consultation with relevant partners, UNDP programme units may decide which outcomes to choose and what modality to use in evaluation. The existing partnerships on the ground, the nature of the programme, planned evaluations by partners and government (so as to seek opportunities for joint evaluations) and other programme-specific factors may influence such decisions. For more details, please refer to the compendium on outcome evaluations.
UNDP programme units may commission evaluations of their respective projects as needed. Managing for results requires, as a starting point, a good knowledge of projects, their effectiveness, internal and external factors affecting effectiveness, their added value and their contribution to higher level outcomes. A project evaluation assesses the performance of a project in achieving its intended results. It yields useful information on project implementation arrangements and the achievement of outputs. It is at this level that direct cause and attribution can be addressed given the close causal linkage between the initiatives and the outputs.
The primary purpose of a project evaluation is to make improvements, to continue or upscale an initiative, to assess replicability in other settings, or to consider alternatives. Therefore, although project evaluations are mandatory only when required by partnership protocols, programme units are strongly recommended to commission evaluations, particularly of pilot programmes, before replication or upscaling, projects that are going into a next phase, and projects more than five years in duration. Increasingly, project evaluations play an important role in accountability to donors and governments involved in financing projects. For their own accountability reasons, donor agencies and other cost-sharing partners35 may request UNDP to include evaluation requirements in the UNDP-donor partnership agreements. Mid-term and final evaluations of Global Environment Facility projects are examples of project evaluations, as they are carried out within the clearly defined scope of a single project.36
When a project is undertaken in partnership with other development actors, the evaluation needs to take into consideration the objectives, inputs and contributions by each partner. The overall evaluation conclusions need to highlight how these different elements integrate to achieve the intended outputs, and what can be learned from the added value of the collaboration. Therefore, it is of central importance that UNDP and the partners involved in a project work together, voice their expectations and issues, and own the evaluation from the planning phase throughout the whole process.
There are several important differences between project evaluations and outcome evaluations, as illustrated in Table 22.
|
Project Evaluation |
Outcome Evaluation |
Focus |
Generally speaking, inputs, activities and outputs (if and how project outputs were delivered within a sector or geographic area and if direct results occurred and can be attributed to the project). (See note.) | Outcomes (whether, why and how the outcome has been achieved, and the contribution of UNDP to a change in a given development situation). |
Scope |
Specific to project objectives, inputs, outputs and activities. |
Broad, encompassing outcomes and the extent to which programmes, project, soft assistance, partners’ initiatives and synergies among partners contributed to its achievement. |
Purpose |
Project based, to improve implementation, to re-direct future projects in the same area or to allow for upscaling of project. |
To enhance development effectiveness, to assist decision-making, to assist policy making, to re-direct future UNDP assistance, to systematize innovative approaches to sustainable human development. |
Source: UNDP, Guidelines for Evaluators, 2002 |
||
Note: Large projects may have outcomes that can be evaluated. Further, small projects may also make tangible contributions to the achievement of CPD outcomes or even project-specific outcomes. In such instances, these project evaluations may be considered to be fulfilling requirements for outcome evaluations.
The increasing focus on outcome evaluations in UNDP does not mean that outcome evaluations have replaced project evaluations. Many programme units continue to undertake project evaluations because they yield useful information on project implementation arrangements, administrative structures and the achievement of outputs. Further, project evaluation provides a basis for the evaluation of outcomes and programmes, as well as for programme and thematic evaluations conducted by the Evaluation Office, and for distilling lessons from experience for learning and sharing knowledge.
To ensure the relevance and effective use of evaluation information, evaluations should be made available in a timely manner so that decision makers can make decisions informed by evaluative evidence.
In addition to project and outcome evaluations, senior managers of programme units may choose to commission thematic evaluations to assess UNDP performance in areas that are critical to ensuring sustained contribution to development results. They may focus on one or several cross-cutting themes that have significance beyond a particular project or initiative. Examples of thematic evaluations commissioned by programme units include the evaluation of UNDP initiatives in a particular results area, such as democratic governance, and the evaluation of a cross-cutting theme, such as capacity development or gender mainstreaming in UNDP programming in a given country.
Country offices may commission a country programme evaluation to assess UNDP attainment of intended results and contributions to national development results in a given country. The evaluation examines key issues that are similar to those in the ADRs, such as UNDP effectiveness in delivering and influencing the achievement of development results and UNDP strategic positioning. The country programme evaluation contributes to the greater accountability of UNDP and the quality assurance of UNDP initiatives at the country level. As in the ADR, it allows findings and recommendations to feed into the preparation of subsequent programmes. It can be used to facilitate dialogue with the government and other national partners and may also provide lessons that are useful for the government in its aid management work and its relationship with other development partners. Despite a number of similarities, country programme evaluations commissioned by country offices are distinct from the ADRs in terms of their scope and management arrangements. They are usually focused on a given programme cycle with a greater focus on performance at the project level. Further, decentralized country programme evaluations are commissioned by those responsible for programme management, as opposed to the independent Evaluation Office.
Similarly, regional bureaux and policy and practice units may decide to carry out mid-term evaluations of their respective regional and global programmes. These mid-term programme evaluations allow for mid-course adjustment of programmes and also feed into the regional and global programme evaluations that the Evaluation Office is mandated to conduct towards the end of the programme period.
An impact evaluation is an evaluation of the effects—positive or negative, intended or not—on individual households and institutions, and the environment caused by a given development activity such as a programme or project. Such an evaluation refers to the final (long-term) impact as well as to the (medium-term) effects at the outcome level.
By identifying if development assistance is working or not, impact evaluation also serves the accountability function. Hence, impact evaluation is aligned with RBM and monitoring the contribution of development assistance toward meeting the MDGs. An impact evaluation is useful when:
Impact evaluation does not simply measure whether objectives have been achieved or assess direct effects on intended beneficiaries. It includes the full range of impacts at all levels of the results chain, including ripple effects on families, households and communities, on institutional, technical or social systems, and on the environment. In terms of a simple logic model, there can be multiple intermediate (short and medium term) outcomes over time that eventually lead to impact—some or all of which may be included in an evaluation of impact at a specific moment in time.
This definition emphasizes the need for understanding the consequences of development initiatives in the longer term. Another important issue connected to impact evaluation is attribution—that is, determining to what extent an initiative, rather than other external factors, has contributed to observed impacts. There are many methods that can be applied to deal with the attribution issue. It is important that this issue be taken into account in the design of the initiative, as well as the evaluation ToR and design.37
UNDP programmes, projects and operations operate in concert to support UNDAF objectives and outcomes that address national priorities. The UNDAF describes the collective response of all UN operations in a country. While UNDP outcome evaluations focus on CPD outcomes, UNDAF evaluations focus on UNDAF outcomes, their contributions to national priorities and the coherence of UNCT support. The UNDAF evaluation is timed to provide inputs to the preparation of the next UNDAF, country programmes and projects by individual agencies. The UNDAF evaluation should take place at the beginning of the penultimate year of the programme cycle and build on UNDAF annual reviews as well as major studies and evaluations that have been completed by individual agencies. Although the results of the UNDAF evaluation are meant to contribute to managing for results, it is an external function, which should be separated from programme management. UNDAF monitoring and evaluation should always be aligned with existing national monitoring and evaluation systems or focus on their development and institutionalization if they are premature or absent.
The scope of the UNDAF evaluation depends on the previous evaluations and studies already conducted during the cycle and on the nature of UNCT operations in a country. UNDAF evaluations are jointly commissioned and managed by the heads of UN organizations and national governments. They are conducted by external consultants selected by mutual agreement between the United Nations and the government through a transparent and thorough selection process. The 2007 CCA and UNDAF Guidelines39 should be consulted for more information.
Evaluations defined by the initiative modality of development initiatives or level of results can be further defined by the timing. For example, a programme unit may undertake a final project evaluation or a mid-term UNDAF evaluation.
Real time evaluations are often undertaken at an early stage of an initiative to provide managers with timely feedback in order to make an immediate difference to the initiative. They are commonly applied in humanitarian or post-conflict contexts to provide implementing staff with the opportunity to analyse whether the initial response or recovery is appropriate in terms of desired results and process. They can also be used in crisis settings where there may be constraints in conducting lengthier evaluations. These constraints include the absence of baseline data, limited data collection efforts due to a rapid turnover of staff members (for example, lack of institutional memory) and difficulty conducting interviews and surveys due to security issues.
Joint evaluation is one modality of carrying out an evaluation to which different partners contribute. Any evaluation can be conducted as a joint evaluation. Increasingly, UNDP is engaged in joint evaluations and there are various degrees of ‘jointness’ depending on the extent to which individual partners cooperate in the evaluation process, merge their evaluation resources and combine their evaluation reporting.40
The joint evaluation approach became popular in the 1990s with the promotion of the approach through the DAC Principles for Evaluation of Development Assistance, which stated, “Joint donor evaluation should be promoted in order to improve understanding of each others’ procedures and approaches and to reduce the administrative burden on the recipient.”41 The Paris Declaration also reinforced the joint evaluation approach through the commitment made by development agencies and partner countries to find more effective ways of working together.42 Joint evaluations can be characterized by a number of benefits and challenges as shown in Box 29.
Challenges
At the country level, one of the most obvious examples of a joint evaluation is the UNDAF evaluation, in which a number of UN organizations and the government participate. In addition, a UNDP country office may jointly carry out, together with the partner government or with a donor, a joint outcome evaluation that looks where both parties are mutually and equally responsible for the evaluation exercise. For guidance on how to organize and manage a joint evaluation process, see Chapter 6.
34. Meta-evaluation is an evaluation of evaluations. It uses findings from a series of evaluations and requires a robust quality assurance mechanism to ensure that the evaluations used as secondary data are credible and of good quality.
35. See section on mandatory evaluations on page 145.
36. The Global Environment Facility, ‘Monitoring and Evaluation Policy’, February, 2006, Available at: http://www.undp.org/gef/05/documents/me/GEF_ME_Policies_and_Precedures_06.pdf
37. Further materials on impact evaluation can be found on the World Bank website sections on Impact Evaluation (www.worldbank.org/impactevaluation) and Network of Networks on Impact Evaluation (www.worldbank.org/ieg/nonie/).
38. UNDG, ‘CCA/UNDAF Monitoring and Evaluation Guidelines’, 2007. Available at: http://www.undg.org/index.cfm?P=259.
40. OECD, ‘Glossary of Key Terms in Evaluation and Results-based Management’, Development Assistance Committee (DAC), Paris, France, 2002. Available at: http://www.oecd.org/dataoecd/29/21/2754804.pdf.
42. OECD, ‘DAC Guidance for Managing Joint Evaluations’, Development Assistance Committee (DAC), Paris, France, 2006. Available at: http://www.oecd.org/dataoecd/28/14/37484787.pdf.
UNDP support and services consist of programmes, projects, partnerships and ‘soft assistance’ such as advocacy, policy advice and coordination support, which may or may not be delivered within a project framework. Programmes and projects have results frameworks that detail the results map and intended results at the output and outcome levels. Evaluations in UNDP are carried out to adequately cover this wide range of UNDP initiatives in order to assess their worth and merit and support the organization’s learning efforts and accountability. The architecture of evaluation in UNDP, therefore, corresponds to the UNDP programmatic structure and its components.