Handbook on Planning, Monitoring and Evaluating
for Development Results

7.4 Focusing the evaluation

Evaluation scope

The evaluation scope narrows the focus of the evaluation by setting the boundaries for what the evaluation will and will not cover in meeting the evaluation purpose. The scope specifies those aspects of the initiative and its context that are within the boundaries of the evaluation. The scope defines, for example:

• The unit of analysis to be covered by the evaluation, such as a system of related programmes, polices or strategies, a single programme involving a cluster of projects, a single project, or a subcomponent or process within a project
• The time period or phase(s) of the implementation that will be covered
• The funds actually expended at the time of the evaluation versus the total amount allocated
• The geographical coverage
• The target groups or beneficiaries to be included

The scope helps focus the selection of evaluation questions to those that fall within the defined boundaries.

Evaluation objectives and criteria 

Evaluation objectives are statements about what the evaluation will do to fulfill the purpose of the evaluation. Evaluation objectives are based on careful consideration of: the types of decisions evaluation users will make; the issues they will need to consider in making those decisions; and what the evaluation will need to achieve in order to contribute to those decisions. A given evaluation may pursue one or a number of objectives. The important point is that the objectives derive directly from the purpose and serve to focus the evaluation on the decisions that need to be made.

Evaluation criteria help focus evaluation objectives by defining the standards against which the initiative will be assessed. UNDP evaluations generally apply the following evaluation criteria to help focus evaluation objectives: relevance, effectiveness, efficiency and sustainability of development efforts.⁵⁰

Relevance concerns the extent to which a development initiative and its intended outputs or outcomes are consistent with national and local policies and priorities and the needs of intended beneficiaries. Relevance also considers the extent to which the initiative is responsive to UNDP human development priorities of empowerment and gender equality issues. Relevance concerns the congruency between the perception of what is needed as envisioned by the initiative planners and the reality of what is needed from the perspective of intended beneficiaries. It also incorporates the concept of responsiveness—that is, the extent to which UNDP was able to respond to changing and emerging development priorities and needs in a responsive manner.

An essential sub-category of relevance is the criteria of appropriateness, which concerns the cultural acceptance as well as feasibility of the activities or method of delivery of a development initiative. While relevance examines the importance of the initiative relative to the needs and priorities of intended beneficiaries, appropriateness examines whether the initiative as it is operationalized is acceptable and is feasible within the local context. For example, an initiative may be relevant in that it addresses a need that intended beneficiaries perceive to be important, but inappropriate because the method of delivery is incongruent with the culture or not feasible given geographic or other contextual realities.  In applying the criterion of relevance, evaluations should explore the extent to which the planning, design and implementation of initiatives takes into account the local context.

Effectiveness is a measure of the extent to which the initiative’s intended results (outputs or outcomes) have been achieved or the extent to which progress toward outputs or outcomes has been achieved.

Evaluating effectiveness in project evaluations involves an assessment of cause and effect—that is, attributing observed changes to project activities and outputs, for example, the extent to which changes in the number of voters can be attributed to a voter education project. Assessing effectiveness in outcome evaluations will more likely examine UNDP contributions toward intended outcomes. For example, an outcome evaluation might explore the extent to which the observed outputs from a voter education project, along with other UNDP-supported outputs such as professionalizing the electoral administration, contributed towards achieving stated outcomes relating to inclusive participation measured by international observers and other reputable experts.

Assessing effectiveness involves three basic steps:
1. Measuring change in the observed output or outcome
2. Attributing observed changes or progress toward changes to the initiative (project evaluation) and determining UNDP contributions toward observed changes
3. Judging the value of the change (positive or negative)

Efficiency measures how economically resources or inputs (such as funds, expertise and time) are converted to results. An initiative is efficient when it uses resources appropriately and economically to produce the desired outputs. Efficiency is important in ensuring that resources have been used appropriately and in highlighting more effective uses of resources.

As the nature and primary purposes of project and outcome evaluations differ, the application of criterion will also differ. For example, in assessing efficiency, a project evaluation might explore the extent to which resources are being used to produce the intended outputs and how resources could be used more efficiently to achieve the intended results. An outcome evaluation may involve estimates of the total UNDP investment (all projects and soft assistance) toward a given development outcome. The application of this criterion, particularly in UNDP outcome evaluations, poses a challenge as the nature of UNDP initiatives (for example, soft assistance), do not always lend themselves to conventional efficiency indicators. In such cases, some analysis of delivery rates, the reasons some initiatives are implemented more quickly than others, and overall management ratios at the programme level might be considered. It is also important to assess how the partnership strategy has influenced the efficiency of UNDP initiatives through cost-sharing measures and complementary activities.

Sustainability measures the extent to which benefits of initiatives continue after external development assistance has come to an end. Assessing sustainability involves evaluating the extent to which relevant social, economic, political, institutional and other conditions are present and, based on that assessment, making projections about the national capacity to maintain, manage and ensure the development results in the future.

For example, an assessment of sustainability might explore the extent to which:

• A sustainability strategy, including capacity development of key national stakeholders, has been developed and implemented.
• There are financial and economic mechanisms in place to ensure the ongoing flow of benefits once the assistance ends.
• Suitable organizational (public or private sector) arrangements have been made.
• Policy and regulatory frameworks are in place that will support continuation of benefits.
• The requisite institutional capacity (systems, structures, staff, expertise, etc.) exists.

Impact measures changes in human development and people’s well-being that are brought about by  development initiatives, directly or indirectly, intended or unintended. Many development organizations evaluate impact because it generates useful information for decision making and supports accountability for delivering results. At times, evaluating impact faces challenges: confirming whether benefits to beneficiaries can be directly attributed to UNDP support can be difficult, especially when UNDP is one of many contributors. However, the impact of UNDP initiatives should be assessed whenever their direct benefits on people are discernible. 

In general, applying the following most commonly applied criteria—relevance, effectiveness, efficiency, sustainability and impact—in combination will help to ensure that the evaluation covers the most critical areas of the initiative. However, not all criteria are applicable, or equally applicable, to every evaluation. Different criteria may need to be applied in unique cases. In determining which criteria to apply, consider the type of evaluation and the contributions of the information to the purpose relative to the cost (use of evaluation resources). For example, evaluations of humanitarian and conflict programming may additionally apply the criteria of connectedness, coherence, coverage and coordination⁵¹. Box 39 outlines guiding questions to help define evaluation criteria and associated evaluation questions.

Evaluation questions

Evaluation questions, when answered, can give users of the evaluation the information they seek in order to make decisions, take action or add to the knowledge base. The evaluation questions refine the focus of the evaluation by making explicit the aspects of the initiative that will be considered when judging its performance.

Evaluation questions reflect the underlying chain of assumptions about how the initiative is expected to operate within its contexts pursuant to the intended outputs and outcomes. The questions chosen for an evaluation should follow from a thorough understanding of the initiative’s operations, intentions and context and should be selected for their role in meeting the evaluation purpose, objectives and relevant evaluation criteria.

An indefinite number of questions could be asked for each evaluation criterion. Real world evaluations are limited in terms of time, budget and resources. Therefore, it is important to be strategic in determining what information is needed most and to prioritize evaluation questions. It is better to answer fewer questions robustly than to answer more superficially. A clear and concise set of the most relevant questions ensures that evaluations are focused, manageable, cost-efficient and useful.
  
To ensure that the key questions selected for the evaluation are the most relevant and most likely to yield meaningful information for users, UNDP programme units must solicit input from and negotiate agreement among partners and other stakeholders, including the evaluation team. Commissioning offices should ensure that the evaluation matrix in the evaluation inception report makes clear the linkages among the evaluation criteria, the evaluation questions and the information needs of intended users (see Annex 3 for more details).

Gender, exclusion sensitivity and rights-based approach

Consistent with UNDP development efforts, UNDP evaluations are guided by the principles of gender equality, the rights-based approach and human development.⁵²Thus, as appropriate, UNDP evaluations assess the extent to which UNDP initiatives: have addressed the issues of social and gender inclusion, equality and empowerment; contributed to strengthening the application of these principles to various development efforts in a given country; and incorporated the UNDP commitment to rights-based approaches and gender mainstreaming in the initiative design.

Mainstreaming a gender perspective is the process of assessing the implications for women and men of any planned action, including legislation, policies or programmes, in all areas and at all levels. It is a strategy for making gender-related concerns and experiences an integral dimension of the design, implementation, monitoring and evaluation of policies and programmes in all political, economic and societal spheres so that women and men benefit equally and inequality is not perpetuated. UNDP evaluations should assess the extent to which UNDP initiatives have considered mainstreaming a gender perspective in the design, implementation and outcome of the initiative and if both women and men can equally access the initiative ’s benefits to the degree they were intended. Similarly, evaluations should also address the extent to which UNDP has advocated for the principle of equality and inclusive development, and has contributed to empowering and addressing the needs of the disadvantaged and vulnerable populations in a given society.

The rights-based approachin development efforts entails the need to ensure that development strategies facilitate the claims of rights-holders and the corresponding obligations of duty-bearers. This approach also emphasizes the important need to address the immediate, underlying and structural causes for not realizing such rights. The concept of civic engagement, as a mechanism to claim rights, is an important aspect in the overall framework. When appropriate, evaluations should assess the extent to which the initiative has facilitated the capacity of rights-holders to claim their rights and duty-bearers to fulfill their obligations.

Evaluations should also address other cross-cutting issues, depending on the focus of the evaluation, such as the extent to which UNDP has incorporated and fostered South-South cooperation, knowledge management, volunteerism and UN reform in its initiative.