CHAPTER 10. COUNTRY PORTFOLIO EVALUATION
A country portfolio evaluation is an assessment of the portfolio of development interventions supported by UNDP and the funds in association with UNDP in a particular country in any given five-year period. The portfolio represents a broad spectrum of initiatives that include programmes, projects, advocacy and other development-related activities. The five-year period may or may not correspond to the programming cycle.
This type of evaluation aims to draw lessons from experience that can be the basis for instituting improvements in the development initiatives supported by UNDP and the funds in association with it. Such lessons include those that pertain to the formulation, implementation and management of development programmes and projects and the conduct of other development-oriented activities such as advocacy and networking among development partners.
The main stakeholders of the evaluation are the Government, UNDP and the funds in association with UNDP. They are interested in knowing the results of their cooperation as a basis for instituting necessary improvements. The cross-fertilization of lessons drawn from country portfolio evaluations in various programme countries can improve the way UNDP and the funds conduct their business - both programming and non-programming work.
Other stakeholders include the private sector, CSOs and donors. The private sector and CSOs are directly involved in, and/or affected by, interventions supported by UNDP and the funds in association with it. The donors want to ensure that resources are used as efficiently as possible to reach target groups.
The evaluation sample should include programmes, projects and activities that have been carried out over a period long enough to offer lessons from experience. It will vary depending on the timing of the evaluation. For example, if the country portfolio evaluation covers a five-year period that corresponds to the CCF period, then the evaluation sample should include those programmes, projects and activities that: (a) were carried over from an earlier period and completed during the CCF period, and (b) were started and completed (or almost completed) during the CCF period.
PLANNING AND MANAGEMENT
A country portfolio evaluation may be under-taken on the initiative of a UNDP country office, a regional bureau, OESP and/or funds in association with UNDP. The Programme Management Oversight Committee (PMOC) may also call for the conduct of the evaluation. Since the evaluation also covers the interventions and activities of funds in association with UNDP, it may be conducted as a joint evaluation by UNDP and the funds concerned.
Regardless of who initiated the evaluation, the UNDP country office must be involved in its planning and organization and must consult the Government during the process.
The evaluation itself must be conducted by a team of independent consultants.
A country portfolio evaluation may be undertaken in any year since it is not confined or linked to a specific programming period.
The time required for the evaluation will vary depending on the size and complexity of the country portfolio being evaluated
For planning details, see chapter 23.
CHAPTER 11. MONITORING AND EVALUATION AND THE UNDP AREAS OF THEMATIC FOCUS
Monitoring and evaluation must provide assess-ments of three dimensions of all programmes and projects: relevance, performance and success.
It is also useful to identify the key elements that are specific to programmes and projects with a particular thematic focus in order to monitor and evaluate those programmes and projects effectively. Some of these key elements are described below. Their assessment must be undertaken within the overall framework for monitoring and evaluation described in the preceding chapters, e.g., use of specific criteria of relevance, performance and success and use of appropriate indicators. Moreover, since the areas of thematic focus are themselves interrelated, the key elements should be assessed in an interrelated, holistic manner.
POVERTY ERADICATION AND SUSTAINABLE LIVELIHOODS
Poverty eradication and the promotion of sustainable livelihoods constitute the top priority of UNDP. UNDP follows a two-pronged strategy to achieve the sustainable eradication of poverty and the shift from poverty to equity:8
Although poverty eradication interventions span a wide range of different types of programmes and projects, monitoring and evaluation of those interventions should always provide assessments of the following elements:
GENDER CONSIDERATIONS IN DEVELOPMENT
Gender-mainstreaming is a strategy for promoting gender equality. It is defined as taking account of the social, cultural, economic and political inequality between women and men in all policy, programming, administrative, human resource and financial activities and all organizational procedures for the empowerment of women. In practice, this means:
The Panel on Monitoring and Evaluation of the ACC Sub-Committee on Rural Development has initiated the development of core evaluation guidelines on gender issues.10 Such guidelines, which may be used for both monitoring and evaluation, emphasize the following:
SUSTAINABLE ENVIRONMENT AND SOUND MANAGEMENT OF NATURAL RESOURCES
Some of the key elements that should be assessed when monitoring and evaluating environmental programmes and projects are listed below.11
Environmental Context of the Programme or Project
Environmental Management Strategies
Integration of an Environmental Programme or Project into the Overall Development Programme of the Area (i.e., local, national, regional or global)
The following key elements of the framework for UNDP partnerships in good governance for SHD12 must be examined:
Characteristics of Sound Governance
Economic, Political and Administrative Governance
Constructive Interaction among the State, the Private Sector and CSOs
11UNDP, BPPE, Handbook and Guidelines for Environmental Management and Sustainable Development, 1992.
12UNDP, BPPS, Governance for Sustainable Human Development (A UNDP Policy Document), January 1997.
CHAPTER 12. INDICATORS
Chapter 12 focuses on indicators for the monitoring and evaluation of UNDP-assisted programmes and projects. Indicators of performance for corporate planning within UNDP are outside the scope of the present document.
Indicators constitute a critical component of a results-oriented monitoring and evaluation framework. Generally speaking, they are signs that show changes in certain conditions or results from specific interventions. They provide evidence of the progress of programme or project activities in the attainment of development objectives.
Within the context of UNDP programmes and projects, indicators are pre-established signs that the people carrying out monitoring and evaluation actions look for in determining the extent to which a programme or project remains relevant, is performing well and is achieving its objectives. In this context, they may be classified as relevance, performance and success indicators.13
SELECTION OF INDICATORS
Indicators for use in monitoring and evaluation should be selected during the formulation stage of a programme or project when the objectives are being established. The following questions should be answered as part of the process of establishing the indicators.
Guiding Principles for the Selection of Indicators
Management Responsibility and Involvement of Stakeholders
At the programme or project formulation stage, the institutions that will be directly responsible for the programme or project and other stakeholders should be involved in selecting a preliminary list of indicators. During implementation, the indicators should be revised in accordance with changes in the programme or project context and design through consensus of the various stakeholders. The participatory process is intended to promote ownership of, and responsibility for, the planned results of the programme or project.
Quantitative, Qualitative and Proxy Indicators
Both quantitative and qualitative indicators should be selected based on the nature of the particular aspect of the programme or project that is being assessed. Efficiency, for instance, lends itself easily to quantitative indicators. On the other hand, measuring dynamic sustainability, which involves people's adaptability to a changing environment, necessitates some qualitative assessment of attitudes and behaviours. However, there are some methodologies, such as beneficiary assessment, rapid rural appraisal (RRA) and structured interviews, that can be used to convert qualitative indicators into quantitative indicators.
In cases where cost, complexity and/or timeliness of data collection prevents a result from being measured directly, proxy indicators may be used to reveal performance trends and make managers aware of potential problems or areas of success. For example, in an environmental protection programme where a target result is the improve-ment in the health of certain lakes, the level of toxins in duck eggs may serve as a proxy indicator of that improvement.14
Limiting the Number of Indicators
A good balance should be achieved between theory and practice, i.e., between what should be and what can be measured. An ideal set of indicators includes indicators of relevance, performance and success. However, for practical purposes, it is absolutely necessary that a thorough selection process be undertaken, through negotiation among the various stakeholders, to arrive at a realistic number of meaningful indicators. The major considerations in selecting the indicators are:
CHARACTERISTICS OF GOOD INDICATORS
A popular code for remembering the characteristics of good indicators is SMART.
Source: ITAD, Monitoring and the Use of Indicators, consultancy report to DG VIII, European Commission, Brussels, 1996.
Collection and Analysis of Data for Indicators
Based on selected indicators, time-series data must be collected and analysed during and after programme or project implementation to support monitoring and evaluation. At the programme or project formulation stage, the following elements must be defined:
The time-series data must be compared with the baseline data (constructed at the formulation stage) describing the problems to be addressed by the programme or project. The data comparison will enable programme or project managers and other key stakeholders to assess whether the programme or project is achieving its objectives.
USES OF INDICATORS
In general, indicators must be incorporated into various stages of the programme or project cycle. At the design or formulation stage, indicators (even those that are tentative) must be established to help to clarify the logical framework of the programme or project. The indicators selected should then be used during programme or project implementation as part of the monitoring process to measure progress, including the identification of potential problems or success. Finally, they should be part of evaluations to assess results, including beneficiary satisfaction with results.
Specifically, indicators must be integrated into the rating and reporting system for monitoring and evaluation purposes (see chapters 13 and 14).
See chapter 20 for guidance on the selection of indicators based on an actual UNDP-assisted programme.