2.4 Finalizing the results framework (deliverable eight)

At this stage of the process, stakeholders should be ready to begin converting the results map into a results framework. In many situations, a smaller group of stakeholders are engaged in this undertaking. However, the wider group can participate in preparing a rough draft of the framework, using simple techniques and without going into the details and mechanics of RBM terminologies.

Creating the draft results framework

Table 5 provides a starting point for converting the results map into a draft framework for UNCT and UNDP programme and project documents. It shows how to translate some of the general terms and questions used in the planning session into the common programming language used by UNCTs and UNDP. The table can be used to produce an initial draft of the results framework with all or most stakeholders. It can be particularly helpful at the project level or in situations involving a diverse group of stakeholders.

Table 5. Rough guide for creating an initial draft of the results framework

Questions and General Terminologies

Equivalent UNDP RBM Terminology

Terms such as: vision, goal, objective, longer term outcome, long-term results
Questions such as: What are we trying to achieve? Why are we working on this problem? What is our overall goal?

Impact

Terms such as: first positive result or immediate result, prerequisites, short- and medium-term results
Questions such as: Where do we want to be in five years? What are the most immediate things we are trying to change? What are the things that must be in place first before we can achieve our goals and have an impact?

Outcome

Terms such as: interventions, programmes
Questions such as: What are the things that need to be produced or provided through projects or programmes for us to achieve our short- to medium-term results? What are the things that different stakeholders must provide?

Outputs

Terms such as: actions
Questions such as: What needs to be done to produce these outputs?

Activities

Terms such as: measure, performance measurement, performance standard
Questions such as: How will we know if we are on track to achieve what we have planned?

Indicators17

Terms such as: data sources, evidence
Questions such as: What precise information do we need to measure our performance?  How will we obtain this information? How much will it cost?  Can the information be monitored?

Means of verificatio17


Formulating strong results and indicators

Having a smaller group of persons with greater familiarity with RBM terminologies usually helps when undertaking this task. This is because it may be difficult with a large group to make progress given the technicalities involved in articulating the results framework. However, when using a smaller group, the information should be shared with the wider group for review and validation. In doing this, exercise stakeholders should:

  • Use a version of Table 6.
  • Refer to the guidance below on formulating the various components of the framework.
  • Complete a table for each major result. Each major result (outcome) may have one or more related impacts. The expected impacts should be filled in for each major result (outcome). Likewise each outcome will have one or more outputs and so forth.
Table 6. The results framework

Results

Indicators

Baseline

Target

Means of Verification

Risks & Assumptions

Impact statement
(Ultimate benefits for target population)

Measure of progress against impact

 

 

 

Assumptions made from outcome to impact. Risks that impact will not be achieved.

Outcome statement
(Short to medium term change in development situation)

Measure of progress against outcome

 

 

 

Assumptions made from outputs to outcome. Risks that outcome will not be achieved.

Outputs
(Products and services- tangible/intangible- delivered or provided)

Measure of progress against output

 

 

 

Assumptions made from activities to outputs. Risks that outputs may not be produced.

Activities
(Tasks undertaken in order to produce research outputs)

Milestones or
key targets for production of outputs

 

 

 

Preconditions for implementation of activities.

Good quality results—that is, well formulated impacts, outcomes, outputs, activities and indicators of progress—are crucial for proper monitoring and evaluation. If results are unclear and indicators are absent or poorly formulated, monitoring and evaluating progress will be challenging, making it difficult for staff and managers to know how well plans are progressing and when to take corrective actions.

The RBM terms used in this section are the harmonized terms of the UNDG, and are in line with the Organisation for Economic Co-operation and Development-Development Assistance Committee (OECD-DAC) definitions.

Results and results chain

The planning exercise up to this point should have led to the creation of many results and an overall results map. These results and the results map can be converted into a results chain and results framework using the standard RBM approach and terminologies.

First, a ‘result’ is a defined as a describable or measurable development change resulting from a cause-and-effect relationship. Different levels of results seek to capture different development changes. The planning exercise (see Section 2.2) led to the creation of various results that were labeled as visions, effects, results, pre-conditions, prerequisites, interventions and so on. In the traditional RBM approach, these results are linked together into what is commonly referred to as a results chain. The results chain essentially tells us what stakeholders want to achieve, why they want to achieve it and how they will go about it. This is not very different from the results map. Now we will convert those results into more specific RBM language and begin to add performance measures to them.

As shown in the draft results framework (Table 5), the vision and longer term goals developed in the results mapping exercise are the impacts that will appear in the results framework, the immediate positive results and some of their pre-conditions and prerequisites will appear as outcomes, lower-level prerequisites will appear as outputs, and so on. These can be shown in the format of a results chain where the lowest level prerequisites are labeled as inputs and the highest as impacts, as illustrated in Figure 9.

Formulating the impact statement

Impacts are actual or intended changes in human development as measured by people’s well-being. Impacts generally capture changes in people’s lives.

The completion of activities tells us little about changes in development conditions or in the lives of people.  It is the results of these activities that are significant. Impact refers to the ‘big picture’ changes being sought and represents the underlying goal of development work. In the process of planning, it is important to frame planned interventions or outputs within a context of their desired impact. Without a clear vision of what the programme or project hopes to achieve, it is difficult to clearly define results. An impact statement explains why the work is important and can inspire people to work toward a future to which their activities contribute.

Similar to outcomes, an impact statement should ideally use a verb expressed in the past tense, such as ‘improved’, ‘strengthened’, ‘increased’, ‘reversed’ or ‘reduced’. They are used in relation to the global, regional, national or local social, economic and political conditions in which people live. Impacts are normally formulated to communicate substantial and direct changes in these conditions over the long term—such as reduction in poverty and improvements in people’s health and welfare, environmental conditions or governance. The MDG and other international, regional and national indicators are generally used to track progress at the impact level. 

Using the example from the results map (Section 2.3, step 5), some of the longer term impacts could be “increased public participation in national and local elections, particularly by women, indigenous populations and other traditionally marginalized groups” and “strengthened democratic processes and enhanced participation by all citizens in decisions that affect their lives.” These impacts would be part of the broader vision of a more vibrant and democratic society.

Formulating the outcome statement

Outcomes are actual or intended changes in development conditions that interventions are seeking to support.

Outcomes describe the intended changes in development conditions that result from the interventions of governments and other stakeholders, including international development agencies such as UNDP. They are medium-term development results created through the delivery of outputs and the contributions of various partners and non-partners. Outcomes provide a clear vision of what has changed or will change globally or in a particular region, country or community within a period of time. They normally relate to changes in institutional performance or behaviour among individuals or groups. Outcomes cannot normally be achieved by only one agency and are not under the direct control of a project manager.  
Since outcomes occupy the middle ground between outputs and impact, it is possible to define outcomes with differing levels of ambition. For this reason, some documents may refer to immediate, intermediate and longer term outcomes, or short-, medium- and long-term outcomes. The United Nations uses two linked outcome level results that reflect different levels of ambition:

  • UNDAF outcomes
  • Agency or country programme outcomes

UNDAF outcomes are the strategic, high-level results expected from UN system cooperation with government and civil society. They are highly ambitious, nearing impact-level change. UNDAF outcomes are produced by the combined effects of the lower level country programme outcomes. They usually require contribution by two or more agencies working closely together with their government and civil society partners.

Country programme outcomes are usually the result of programmes of cooperation or larger projects of individual agencies and their national partners. The achievement of country programme outcomes depends on the commitment and action of partners. 

When formulating an outcome statement to be included in a UNDP programme document, managers and staff are encouraged to specify these outcomes at a level where UNDP and its partners (and non-partners) can have a reasonable degree of influence. In other words, if the national goals reflect changes at a national level, and the UNDAF outcomes exist as higher level and strategic development changes, then the outcomes in UNDP programme documents should reflect the comparative advantage of and be stated at a level where it is possible to show that the UNDP contribution can reasonably help influence the achievement of the outcome. For example, in a situation where UNDP is supporting the government and other stakeholders in improving the capacity of the electoral administration agency to better manage elections, outcomes should not be stated as “improved national capacities” to perform the stated functions, but rather “improved capacities of the electoral administration bodies” to do those functions. “Improved national capacity” may imply that all related government ministries and agencies have improved capacity and may even imply that this capacity is also improved within non-government bodies. If this was indeed the intention, then “improved national capacities” could be an accurate outcome. However, the general rule is that government and UNDP programme outcomes should be fairly specific in terms of what UNDP is contributing, while being broad enough to capture the efforts of other partners and non-partners working towards that specific change.

An outcome statement should ideally use a verb expressed in the past tense, such as ‘improved’, ‘strengthened’ or ‘increased’, in relation to a global, regional, national or local process or institution. An outcome should not be stated as “UNDP support provided to Y” or “technical advice provided in support of Z” but should specify the result of UNDP efforts and that of other stakeholders for the people of that country.

  • An outcome statement should avoid phrases such as “to assist/support/develop/monitor/identify/follow up/prepare X or Y.”
  • Similarly, an outcome should not describe how it will be achieved and should avoid phrases such as “improved through” or “supported by means of.”
  • An outcome should be measurable using indicators. It is important that the formulation of the outcome statement takes into account the need to measure progress in relation to the outcome and to verify when it has been achieved. The outcome should therefore be specific, measurable, achievable, relevant and time-bound (SMART).
  • An outcome statement should ideally communicate a change in institutional or individual behaviour or quality of life for people—however modest that change may be.

The following illustrate different levels of outcomes:

  • Policy, legal and regulatory framework reformed to substantially expand connectivity to information and communication technologies (short to medium term)
  • Increased access of the poor to financial products and services in rural communities (medium to long term)
  • Reduction in the level of domestic violence against women in five provinces by 2014 (medium to long term)
  • Increased volume of regional and sub-regional trade by 2015 (medium to long term)

Using the previous elections example, the outcome at the country programme level may be “enhanced electoral management systems and processes to support free and fair elections” or “electoral administrative policies and systems reformed to ensure freer and fairer elections and to facilitate participation by marginalized groups.”  

Formulating the output statement

Outputs are short-term development results produced by project and non-project activities. They must be achieved with the resources provided and within the time-frame specified (usually less than five years.)

Since outputs are the most immediate results of programme or project activities, they are usually within the greatest control of the government, UNDP or the project manager. It is important to define outputs that are likely to make a significant contribution to achievement of the outcomes. 

In formulating outputs, the following questions should be addressed:

  • What kind of policies, guidelines, agreements, products and services do we need in order to achieve a given outcome?
  • Are they attainable and within our direct control?
  • Do these outputs reflect an appropriate strategy for attaining the outcome? Is there a proper cause and effect relationship?
  • Do we need any additional outputs to mitigate potential risks that may prevent us from reaching the outcome?
  • Is the output SMART—specific, measurable, achievable, relevant and time-bound?

It is important to bear in mind:

  • Outputs must be deliverable within the respective programming cycle.
  • Typically, more than one output is needed to obtain an outcome.
  • If the result is mostly beyond the control or influence of the programme or project, it cannot be an output.

Outputs generally include a noun that is qualified by a verb describing positive change. For example:

  • Study of environment-poverty linkages completed
  • Police forces and judiciary trained in understanding gender violence
  • National, participatory forum convened to discuss draft national anti-poverty strategy
  • National human development report produced and disseminated

Returning to our example, there could be a number of outputs related to the outcome “electoral management policies and systems reformed to ensure freer and fairer elections and to facilitate participation by marginalized groups.” Outputs could include:

  • Advocacy campaign aimed at building consensus on need for electoral law and system reform developed and implemented
  • Systems and procedures implemented and competencies developed in the national electoral management agency to administer free and fair elections
  • Training programme on use of new electoral management technology designed and implemented for staff of electoral management authority
  • Revised draft legislation on rights of women and indigenous populations to participate in elections prepared
  • Electoral dispute resolution mechanism established 
Formulating the activities

Activities describe the actions that are needed to obtain the stated outputs. They are the coordination, technical assistance and training tasks organized and executed by project personnel.

In an RBM context, carrying out or completing a programme or project activity does not constitute a development result. Activities relate to the processes involved in generating tangible goods and services or outputs, which in turn contribute to outcomes and impacts.

In formulating activities the following questions should be addressed:       

  • What actions are needed in order to obtain the output?
  • Will the combined number of actions ensure that the output is produced?
  • What resources (inputs) are necessary to undertake these activities?

It is important to bear in mind:

  • Activities usually provide quantitative information and they may indicate periodicity of the action.
  • Typically, more than one activity is needed to achieve an output.

           
Activities generally start with a verb and describe an activity or action. Using our example, activities could include:

  • Provide technical assistance by experts in the area of electoral law reform
  • Develop and deliver training and professional development programmes for staff
  • Organize workshops and seminars on electoral awareness
  • Publish newsletters and pamphlets on electoral rights of women and minorities
  • Procure equipment and supplies for Electoral Management Authority
  • Engage consultants to draft revised electoral laws
Formulating inputs

Inputs are essentially the things that must be put in or invested in order for activities to take place.

Though not dealt with in detail in this manual, inputs are also part of the results chain. Inputs include the time of staff, stakeholders and volunteers; money; consultants; equipment; technology; and materials. The general tendency is to use money as the main input, as it covers the cost of consultants, staff, materials, and so forth. However, in the early stages of planning, effort should be spent on identifying the various resources needed before converting them into monetary terms. 

The guidance above should help to prepare the first column (‘results’) in the results framework.

Table 7. The ‘results’ sections of the results framework
Results

Impact statement
Ultimate benefits for target population

Outcome statement
Short to medium-term change in development situation
Normally more than one outcome will be needed to attain the impact

Outputs
Products and services (tangible/intangible) delivered or provided
Normally more than one output will be needed to achieve the outcome

Activities
Tasks undertaken in order to produce research outputs
Each output normally has a number of activities



Box 12. Note on results framework
The results framework can be completed with all the outcomes, outputs, activities and inputs that the stakeholders have identified. However, in many cases, a more limited framework showing only the specific outcomes and outputs related to a particular agency (such as UNDP) and its partners will be needed to satisfy internal requirements. In these cases where a more focused results framework is created, every effort should be made to show information on the broader agenda of actions being pursued and the partners and non-partners working towards achieving the overall outcomes and impacts in the narrative of the wider strategy document (such as the UNDAF; the global, regional or country programme action plan; or the project document). The strategy document should not be confined to only what the agency will produce. It should instead show how the efforts of different stakeholders will contribute to the achievement of the common overall vision and intended impacts. This will also aid the monitoring and evaluation processes.

 
Formulating performance indicators

Indicators are signposts of change along the path to development. They describe the way to track intended results and are critical for monitoring and evaluation.

Good performance indicators are a critical part of the results framework. In particular, indicators can help to:

  1. Inform decision-making for ongoing programme or project management
  2. Measure progress and achievements, as understood by the different stakeholders
  3. Clarify consistency between activities, outputs, outcomes and impacts
  4. Ensure legitimacy and accountability to all stakeholders by demonstrating progress
  5. Assess project and staff performance18

Indicators may be used at any point along the results chain of activities, outputs, outcomes and impacts, but must always directly relate to the result being measured.  Some important points include the following:

  • Who sets indicators is fundamental, not only to ownership and transparency but also to the effectiveness of the indicators. Setting objectives and indicators should be a participatory process.
  • A variety of indicator types is more likely to be effective. The demand for objective verification may mean that focus is given to the quantitative or simplistic at the expense of indicators that are harder to verify but may better capture the essence of the change taking place.
  • The fewer the indicators the better. Measuring change is costly so use as few indicators as possible.  However, there must be indicators in sufficient number to measure the breadth of changes happening and to provide cross-checking.

Box 13. Note on performance indicators
A frequent weakness seen in formulating indicators is the tendency to use general and purely quantitative indicators that measure number or percentage of something, for example, “number of new policies passed.” These are often weak indicators as they merely communicate that something has happened but not whether what has happened is an important measure of the objective. For example, take a situation where an audit report finds 10 weaknesses in a business unit, 3 of which are considered serious and the other 7 routine. If the 7 routine issues were dealt with, an indicator that measures performance as “number or percentage of recommendations acted on” may capture the fact that some action has been taken but not convey a sense of whether these are the important actions.
In general, indicators should direct focus to what is critical. For example, there could be different ways of measuring whether an outcome relating to greater commitment by government partners to HIV/AIDS concerns is being realized.
Examine the following indicator:“Number of government ministries that have an HIV/AIDS sector strategy”

Now compare it with another quantitative indicator such as:  “Number of government ministries that have an HIV/AIDS sector strategy developed in consultation with non-governmental stakeholders”

And further compare it with a possible qualitative indicator: “Number of government ministries that have a strong HIV/AIDS sector strategy”

Measured by:

  • Strategy was developed in consultation with non-government stakeholders (X points)
  • Ministry’s senior officials involved in strategy development and implementation processes (Y points)
  • Ministry has in place a budget to finance implementation of strategy (X points)
In the first case, a strategy could have been designed, with no stakeholder involvement, no senior management engagement, and no budget. Simply counting the number of ministries that have done this would not be a measure of real progress against the outcome that deals with the real commitment of the government partners.

The process of formulating indicators should begin with the following questions:

  • How can we measure that the expected results are being achieved?
  • What type of information can demonstrate a positive change?
  • What can be feasibly monitored with given resource and capacity constraints?
  • Will timely information be available for the different monitoring exercises?
  • What will the system of data collection be and who will be responsible?
  • Can national systems be used or augmented?
  • Can government indicators be used? 19

Box 14. SMART indicators
Specific: Is the indicator specific enough to measure progress towards the results?
Measurable: Is the indicator a reliable and clear measure of results?
Attainable: Are the results in which the indicator seeks to chart progress realistic?
Relevant: Is the indicator relevant to the intended outputs and outcomes?

Time-bound: Are data available at reasonable cost and effort?

 
Quantitative and qualitative indicators

Indicators can either be quantitative or qualitative. Quantitative indicators are statistical measures that measure results in terms of:

  • Number
  • Percentage
  • Rate (example: birth rate—births per 1,000 population)
  • Ratio (example: sex ratio—number of males per number of females)

Qualitative indicators reflect people’s judgments, opinions, perceptions and attitudes towards a given situation or subject. They can include changes in sensitivity, satisfaction, influence, awareness, understanding, attitudes, quality, perception, dialogue or sense of well-being.

Qualitative indicators measure results in terms of:

  • Compliance with…
  • Quality of…
  • Extent of…
  • Level of …

Note that in the example used in Box 13 on the commitment of government partners, sub-indicators are being used to assess the quality of the strategy, “Did it benefit from the involvement of other stakeholders?”; the extent of senior management engagement; and the level of commitment, “Is there also a budget in place?”

As far as possible, indicators should be disaggregated. Averages tend to hide disparities, and recognizing disparities is essential for programming to address the special needs of groups such as women, indigenous groups and marginalized populations. Indicators can be disaggregated by sex, age, geographic area and ethnicity, among other things.

The key to good indicators is credibility—not volume of data or precision in measurement. Large volumes of data can confuse rather than bring focus and a quantitative observation is no more inherently objective than a qualitative observation. An indicator’s suitability depends on how it relates to the result it intends to describe.

Proxy indicators

In some instances, data will not be available for the most suitable indicators of a particular result. In these situations, stakeholders should use proxy indicators. Proxy indicators are a less direct way of measuring progress against a result.

For example, take the outcome: “improved capacity of local government authorities to deliver solid waste management services in an effective and efficient manner.” Some possible direct indicators could include:

  • Hours of down time (out of service time) of solid waste vehicle fleet due to maintenance and other problems
  • Percentage change in number of households serviced weekly
  • Percentage change in number of commercial properties serviced weekly
  • Percentage of on-time pick-ups of solid waste matter in [specify] region within last six-month period

Assuming no system is in place to track these indicators, a possible proxy or indirect indicator could be:

  • A survey question capturing the percentage of clients satisfied with the quality and timeliness of services provided by the solid waste management service. (The agency may find it easier to undertake a survey than to introduce the systems to capture data for the more direct indicators.)

The assumption is that if client surveys show increased satisfaction, then it may be reasonable to assume some improvements in services. However, this may not be the case, which is why the indicator is seen as a proxy, rather than a direct measure of the improvement. 

Similarly, in the absence of reliable data on corruption in countries, many development agencies use the information from surveys that capture the perception of corruption by many national and international actors as a proxy indicator.

In the Human Development Index, UNDP and other UN organizations use ‘life expectancy’ as a proxy indicator for health care and living conditions. The assumption is that if people live longer, then it is reasonable to assume that health care and living conditions have improved. Real gross domestic product/capita (purchasing power parity) is also used in the same indicator as a proxy indicator for disposable income.

Levels of indicators

Different types of indicators are required to assess progress towards results. Within the RBM framework, UNDP uses three types of indicators:

  1. Impact indicators
  2. Outcome indicators
  3. Output indicators

Impact indicators describe changes in people's lives and development conditions at the global, regional and national development situation. In the case of community-level initiatives, imapct indicators can describe such changes at the sub-national and community levels. They provide a broad picture of whether the development changes that matter to people and UNDP are actually occurring. In the context of country-level planning (CPD), impact indicators are often at the UNDAF and MDG levels, and often appear in the UNDAF results framework. Impact indicators are most relevant to global, regional and national stakeholders and senior members of the UNCT for use in monitoring. Table 8 includes some examples of impact indicators.

Table 8.  Impact indicators

Sample Impacts

Sample Indicators
(i.e., “What can we see to know if change is happening?”)

  • Increased public participation in national and local elections, particularly by women, indigenous populations and other traditionally marginalized groups
  • Overall proportion of eligible voters who vote in the national (or local) elections
  • Percentage of eligible women who vote in the  elections
  • Percentage of eligible indigenous people who vote in elections
  • Improved educational performance of students in region of the country
  • Percentage of students completing primary schooling
  • Pass rates in standardized student tests
  • Reduction in poverty and hunger
  • Poverty rate
  • Gini coefficient
  • Percentage of population living in extreme poverty
  • Level of infant malnutrition
  • People are healthier and live longer
  • Longevity
  • Infant mortality
  • HIV/AIDS prevalence rate

Outcome indicators assess progress against specified outcomes. They help verify that the intended positive change in the development situation has actually taken place. Outcome indicators are designed within the results framework of global, regional and country programmes. Outcome indicators are most often useful to the UN organization and its partners working on the specific outcome. Table 9 gives a few examples.

Note: Outcome indicators are not intended to only measure what an agency (such as UNDP) does or its contribution. They are indicators of change in development conditions and are therefore expected to be at a higher level than the indicators of the agency’s outputs.

 
Table 9. Outcome indicators

Sample Outcomes

Sample Indicators
(i.e., “What can we see to know if change is happening?”)

  • Electoral administrative policies and systems reformed to ensure freer and fairer elections and to facilitate participation by marginalized groups
  • Percentage of citizens surveyed that believe that the electoral management process is free and fair. (This is a proxy indicator. Instead of a general survey of citizens, a more limited survey could be done of a selected group of persons as well.)
  • Percentage of women and minorities surveyed that are aware of their rights under the new electoral administration laws.
  • Annual percentage increase in number of women registered to vote. (This is an intermediate indicator of progress, getting to the point when the impact indicator of how many of these groups actually vote can be measured.)
  • Annual percentage increase in number of indigenous people registered to vote.
  • Ratio of voter registration centres per population in rural areas.
  • Policy, legal and regulatory framework reformed to substantially expand connectivity to information and communication technologies
  • Number and proportion of the population with access to the Internet, disaggregated by gender. (This could be occurring without the changes to the framework. It is useful to track an indicator of this nature because it goes beyond the immediate result and looks at the impacts that partners are concerned with.)
  • Number of key national policies on information technology that are revised and promulgated. (For example, this could be used where it is known that there are a few specific legislations that need to be reformed.)
  • Improved e-governance capacity of key central government ministries and agencies by 2015
  • Extent to which key central government bodies have strong online facilities for citizen engagement. This is measured by composite indicator totaling a selected number of points:
    • Key central government ministries have websites established (10 points)
    • Websites contain functional contact information (10 points)
    • Websites provide functional access to major government policy documents and publications (10 points)
    • Websites facilitate access to persons with disabilities (or is available in second language) (10 points)
    • Websites provide links to other major government departments (10 points)
    • Websites facilitate online payment for important government services (taxes, motor vehicle registration, etc.) (10 points)
  • Percentage of property tax revenue collected through online payment systems.
  • Reduced levels of corruption in the public sector by 2016
  • Corruption perception index. (This is usually measured by a composite survey indicator of the perceptions of national and international experts and the general population about corruption in the country.)
  • Overall conclusion or rating of government performance in addressing corruption in the Independent Audit Office Annual Report.
  • Reduction in level of violence against women by 2013
  • Number of reported cases of domestic abuse against women.20
  • Percentage of women who feel that violence against women has reduced within the last five years (based on survey).
  • Proportion of men who believe that wife beating is justified for at least one reason (based on survey).

In the second example in Table 9, an indicator on whether policies are being changed is used together with one on number of people with access to the Internet to give a broad and complementary view of overall progress against the outcome. It is often necessary to use a set of complementary indicators to measure changes attributable to reasons other than the intervention in question. Also, composite indicators can be used to provide more qualitative measures of progress. Stakeholders can agree on their own composite indicators in areas where no good composite indicator already exists.

Output indicators assess progress against specified outputs. Since outputs are tangible and deliverable, their indicators may be easier to identify. In fact, the output itself may be measurable and serve as its own indication of whether or not it has been produced. Table 10 includes some examples.

Table 10. Output indicators

Sample Outputs

Sample Indicators
“i.e. “What can we see to know if change is happening?”)

  • Draft new policy on electoral reform formulated and submitted to Cabinet
  • Level of progress made in drafting new policy (see Box 15)
  • National electoral management agency has systems, procedures and competencies to administer free and fair elections
  • Percentage of electoral centres using multiple forms of voter identification measures
  • Number of centres that are headed by trained professional staff
  • Percentage of electoral management office staff and volunteers trained in techniques to reduce voter fraud
  • Percentage of electoral management office staff who believe that their agency is more professional and better run than one year ago
  • District school teachers trained
  • Number of teachers trained by end 2010
  • Percentage of teachers trained that were rated as more effective in doing their jobs one year later*
  • National human development report produced and disseminated
  • Number of copies of National Human Development Report distributed
  • Percentage of parliamentarians who receive copy
  • Extent to which National Human Development Report findings and recommendations were used to inform high-level policy discussions (can be composite indicator that looks at whether there was a discussion in Parliament, Cabinet, Meeting of Social Policy Ministers, etc. to discuss findings) *
  • Civil society and community organizations in  region have resources and skills to contribute to monitoring of local poverty reduction strategies
  • Number of NGO staff completing training courses in poverty analysis by end 2009
  • Percentage of trained NGO staff who feel that they are more effective at doing their jobs one year later*
  • Percentage of districts with Monitoring Committees
  • Percentage of districts with Citizen Community Boards

* These indicators represent result type indicators. It is useful to have at least two indicators for an output: one process indicator that gives an idea as to whether the product or service was completed or delivered, and one result indicator that addresses whether the completed output is bringing about intended changes. In this way, programme and project managers can discuss not only the progress of planned outputs and activities, but the quality and impact of those outputs and activities.

Box 15. Using ‘level of progress made’ as an output indicator
In many situations, people struggle with what type of indicator to use for certain outcomes, particularly where counting numbers of things produced may not be meaningful. In this Handbook, we suggest that for certain complex outputs or those outputs where the quality and not the number of what is produced is most critical, one indicator could be ‘level of progress made’. Targets would be set for the level of progress to be made each year. These level-of-progress indicators can be complemented by client satisfaction indicators assessing the extent to which persons were satisfied with what was produced.

QUICK CHECKLIST FOR REVIEWING OUTCOMES AND OUTCOME INDICATORS

YES

NO

  1. The outcomes and their indicators are specific, measurable, achievable, realistic and time-bound (SMART).

 

 

  1. The outcomes clearly outline an area of work where the agency and its partners can have significant influence.

 

 

  1. The outcomes are worded in such a way that they communicate what has changed, for whom (if relevant) and by when. (Outcomes should generally be achievable within 5 years.)

 

 

  1. The outcomes clearly address the interests and concerns of men, women, and marginalized groups (if relevant.)

 

 

  1. The outcomes address changes in institutional capacities and behavior that should lead to sustainable development of the country or region.

 

 

  1. The outcomes speak to changes in conditions and capacities and not delivery of products and services.

 

 

  1. The outcomes have indicators that signal how the desired change will be measured.

 

 

  1. The outcome indicators are measures of change that go beyond what one agency will produce or deliver. They are measures of change in the country or region and not measures of what projects will produce.

 

 

  1. The outcome and its indicators provide a very clear and precise image or picture of what the future should look like, and is not so general that it could cover almost anything.

 

 


QUICK CHECKLIST FOR REVIEWING OUTPUTS AND OUTPUT INDICATORS

YES

NO

  1. The outputs and their indicators are specific, measurable, achievable, realistic and time-bound (SMART).

 

 

  1. The outputs are defined as products or services made possible by the resources provided in a project.

 

 

  1. The language used to describe the outputs includes the noun or thing to be produced as well as the verb describing what happens on completion of the output.

 

 

  1. The outputs are defined as things over which one or more agencies have control and can be held accountable for delivering.

 

 

  1. The outputs defined are necessary ingredients for achieving the outcomes.

 

 

  1. There are indicators that measure both the process of producing the outputs (e.g. how many of something was done) as well as the quality and/or effect of what was produced (e.g. level of usage or user satisfaction with what was produced).

 

 

 
Baselines and targets

Once the indicators are identified, the stakeholders should establish baselines and targets for the level of change they would like to see. It is often better to have a small group undertake the effort of researching the baseline separately, as stakeholders may not have all the data at the time. The baseline and target should be clearly aligned with the indicator, using the same unit of measurement. (For practical reasons, some indicators may need to be adjusted to align with existing measures, such as national surveys or censuses.)

Baseline data establishes a foundation from which to measure change. Without baseline data, it is very difficult to measure change over time or to monitor and evaluate. With baseline data, progress can be measured against the situation that prevailed before an intervention.21  

Once the baseline is established, a target should be set. The target will normally depend on the programme period and the duration of the interventions and activities. For example, within the context of a UNDAF, targets are normally set as five-year targets so as to correspond with the duration of the UNDAF.  Likewise, global, regional and country programmes will normally have four- to five-year targets. While some development change can take a long time to occur, often 10 years or more, the inclusion of a target for the programme or project cycle is intended to enable stakeholders to look for ‘signs’ of overall change. If targets cannot be set for a four- to five-year period, then the indicator used was probably too high a level, and the team will need to find other indicators of progress within the short to medium term.

At the output level, targets can be set for a much shorter period, such as one year, six months and so forth. Relating this to our indicator examples above, Table 11 gives examples of baselines and targets.

Table 11. Indicators, baselines and targets

IMPACT: Increased public participation in national and local elections, particularly by women, indigenous populations and other traditionally marginalized groups

Indicator

Baseline

Target

  • Overall proportion of eligible voters who vote in the national (or local) elections
  • 2006: 42% of eligible voters voted in national elections
  • 2010: 70% of eligible voters vote in national elections
  • Percentage of eligible women who vote in the elections
  • 2006: 0% voted (women were not allowed to vote)
  • 2010: 50% of eligible women vote in national elections
  • Percentage of eligible indigenous people who vote in elections
  • 2006: 15% voted (no efforts were made to encourage or support voting by indigenous people living in the interior)
  • 2010: 45% of eligible indigenous persons vote in national elections

OUTCOME: Electoral administrative policies and systems reformed to ensure freer and fairer elections and to facilitate participation by marginalized groups.

  • Percentage of public that believe that the electoral management process is free and fair
  • 2006: 30% (based on last survey conducted)
  • 2010: 80%
  • Percentage of women and minorities aware of their rights under the new electoral administration laws
  • 2007: 20% of minorities said they were aware of their rights (survey done by [specify] agency; note: women were not allowed to vote)
  • 2010: 70% of women and minorities aware of their rights
  • Percentage increase in number of women registered to vote
  • 2007: 0% of women registered to vote (women were not allowed to vote)
  • 20% annual increase in percentage of eligible women registered to vote
  • Percentage increase in number of indigenous people registered to vote
  • 2007: 30% of eligible minorities registered to vote
  • 20% annual increase in percentage of eligible minorities registered to vote
  • Ratio of voter registration centres per population in rural areas
  • 2006: 1 centre to 11,000 people
  • 2010: 1 centre to 4,000 people

OUTPUT 1: Draft new policy on electoral reform formulated and submitted to Cabinet

  • Progress made in drafting new policy
  • 2008: Agreement reached between major political parties on need to redraft electoral legislation
  • 2009: 5 major public consultations held and white paper prepared on new policy

OUTPUT 2: National electoral management agency has systems, procedures and competencies to administer free and fair elections

  • Percentage of electoral centres using multiple forms of voter identification measures
  • 2006: 0% of centres used multiple forms of voter identification
  • 2009: 70% of centres use two or more forms of voter identification, including fingerprint identification (annual targets may be set)
  • Number of centres that are headed by trained, publicly recruited  professional staff
  • 2006: 20% of centres were run by publicly recruited professional staff (based on study done by [specify] agency)
  • 2009: 80% of centres run by professional staff recruited through public recruitment process
  • Percentage of electoral management office staff who believe that their agency is more professional and better run than 1 year ago
  • No baseline exists; survey to be introduced for the first time in 2008
  • 2009: 70% of staff believe their agency is more professional and better run than 1 year ago
  • Percentage of electoral management office staff and volunteers trained in techniques to reduce voter fraud
  • 2006: 0%
  • 2009: 80%

It may not always be possible to have a strong or high output indicator target for the first year of implementation. For example, consider indicator in Table 10 : “percentage of electoral management office staff and volunteers trained in techniques to reduce voter fraud.” A number of actions may need to be taken in the first year before training begins in the second year. The target for this indicator could therefore be 0 percentin 2009. This does not mean that the indicator is weak. In situations such as this, a ‘comments’ field could be used to explain the target. This is another reason why having two or more indicators to capture different dimensions of the output is recommended (the same applies to the outcome). In this case, another indicator on “level of progress made” in putting in place basic systems, training materials and so forth could be used in addition to the numeric indicator. This would allow for qualitative targets to be set for each year and could address the things that are to be put in place to form the platform for activities that will occur in future years.

Means of verification

Results statements and indicators should be SMART.. The ‘M’ in SMART stands for ‘measurable’, which implies that data should be readily available in order to ascertain the progress made in achieving results. In defining results and corresponding indicators, it is thus important to consider how data will be obtained through monitoring and evaluation processes.

Means of verification play a key role in grounding an initiative in the realities of a particular setting. Plans that are too ambitious or developed too hastily often fail to recognize the difficulties in obtaining evidence that will allow programme managers to demonstrate the success of an initiative. Without clearly defining the kind of evidence that will be required to ascertain the achievement of results, without fully considering the implications of obtaining such evidence in terms of effort and cost, planners put the integrity of the programme at risk. If results and indicators are not based on measurable, independently verifiable data, the extent to which an initiative is realistic or achievable is questionable.

Identifying means of verification should take place in close coordination with key stakeholders. Evidence on outcomes (let alone impact) will need to be provided by the target group, beneficiaries or development partners. Therefore, it is important that in planning programmes and projects, such stakeholders are involved in thinking about how evidence on progress will be obtained during implementation and after completion of the initiative. Clear means of verification thus facilitates the establishment of monitoring systems and contributes significantly to ensuring that programmes and projects are evaluation-ready.

Table 12. Sample results framework with means of verification

IMPACT: Increased public participation in national and local elections, particularly by women, indigenous populations and other traditionally marginalized groups

Indicator

Baseline

Target

Means of Verification

  • Overall proportion of eligible voters who vote in the national (or local) elections
  • 2006: 42% of eligible voters voted in national elections
  • 2010: 70% of eligible voters vote in national elections
  • Office of Electoral Administration’s final report on elections

OUTCOME: Electoral administrative policies and systems reformed to ensure freer and fairer elections and to facilitate participation by marginalized groups

  • Percentage of public that believe that the electoral management process is free and fair
  • 2006: 30% (based on last survey conducted)
  • 2010: 80%
  • Special survey to be undertaken as part of the electoral assistance project in 2008 and 2010
  • Percentage increase in number of women registered to vote
  • 2007: 0% of women registered to vote (women were not allowed to vote)
  • 20% annual increase in percentage of eligible women registered to vote
  • Office of Electoral Administration’s database
  • Ratio of voter registration centres per population in rural areas
  • 2006: 1 centre to 11,000 people
  • 2010: 1 centre to 4,000 people
  • To be computed based on number of centres (Electoral Office database) in relation to population in rural areas (National Planning Agency’s 2010 demographic survey)

OUTPUT 1: Draft new policy on electoral reform formulated and submitted to Cabinet

  • Progress made in drafting new policy
  • 2008: Agreement reached between major political parties on need to redraft electoral legislation
  • 2009: 5 major public consultations held and white paper prepared on new policy
  • Report from government agency organizing workshops
  • Record of Parliamentary proceedings (for submission of white paper) to be obtained from Office of Public Sector Information

OUTPUT 2: National electoral management agency has systems, procedures and competencies to administer free and fair elections

  • Percentage of electoral centres using multiple forms of voter identification measures
  • 2006: 0% of centres used multiple forms of voter identification
  • 2009: 70% of centres use two or more forms of voter identification, including fingerprint identification (annual targets may be set)
  • Electoral Office database

Based on this guidance, the team of stakeholders should refine or finalize the results framework for either the programme or project being developed.

The formulation of a results framework is a participatory and iterative process. Participation is key to ensuring that stakeholders understand and support the initiative and are aware of the implications of all elements of the results framework. In developing a results framework, the definition of new elements (such as formulating outputs after identifying outcomes, or defining indicators after defining a particular result, or specifying the means of verification after defining indicators) should be used to test the validity of previously defined elements.
Relationship between the results framework and UNDP RBM systems

The data created in the planning exercise may appear at different times in various planning documents and systems. For example:

  • The impacts and national priorities appear in the relevant sections of the UNDAF or global, regional or country programme results framework when these are developed.
  • The impact developed in a global, regional or country programme would also be entered in the RBM platform (home.undp.org) in the global, regional or national goal field.
  • Impact indicators are normally entered in national strategy documents and plans and in the UNDAF results framework. Reference can also be made to these indicators in the situation analysis and statements of objective in a CPD or CPAP.
  • The analysis of what is causing the problems would normally be reflected in the situation analysis section of the respective programme or project document.
  • The analysis of what needs to happen or be in place to achieve the goals and impact would also be reflected in the programme or project document, along with any government or UNDP action needed to influence partners and non-partners to take desired actions. This would be captured in the objectives and strategy sections of the respective documents. 
  • The specific outcomes that UNDP will support would be entered in the relevant sections of the UNDAF.
  • The UNDP outcomes identified in the UNDAF are used to formulate the CPD that is approved by the UNDP Executive Board.
  • The same outcomes (or slightly revised outcomes based on the CPAP process but with the same intention) would be entered into Atlas as part of the programme’s project tree. These outcomes would then appear on the programme planning and monitoring page of the RBM platform.
  • Outcome indicators would be entered in the relevant sections of the programme documents and the same indicators (or slightly revised indicators based on the CPAP refinement process) would be entered into the RBM platform at the start of the programme.
  • Baselines and targets would be entered for the outcome indicators in both places as well.
  • The UNDAF and CPD would normally include a set of outputs that the programme intends to produce.
  • These outputs are normally refined in the CPAP process as stakeholders obtain greater clarity on the implementation details for the programme. This may occur months after the UNDAF or CPD has been finalized.
  • The CPAP outputs would be created as output projects in Atlas, together with their indicators, baselines and targets. This information would then appear in the RBM platform to facilitate monitoring and reporting against these outputs. As far as possible, the project outputs in Atlas should have, as their long description, the same wording as the outputs created in the results framework. Likewise, the indicators and baselines for the outputs are the same as should be entered in Atlas. The output targets are also the same as the annual output targets that are used in Atlas and are normally entered when offices prepare their development work plans and set targets for the year.This is illustrated in Figure 11.
 

  • The risks and assumptions would be documented in the relevant column of the programme results and resources table. The risks would also be entered in Atlas and related to an Award (the Award is a collection of outputs). These would then be reflected in the RBM platform for monitoring purposes.
  • Information on partners would be entered in the results framework, and the programme document would explain the efforts of both partners and non-partners in contributing to the outcomes and impact. The role of partners should be included in the formal monitoring and evaluation process (such as in a joint evaluation of an UNDAF). The efforts of non-partners can be monitored informally through meetings with them or other means.

Atlas and the RBM platform should serve as tools to enter the information contained in the results framework and to conduct transactions and monitor progress. The development work plan component of the platform is therefore a monitoring tool for the global, regional and country programmes (or CPAPs), as it captures the outcomes, outcome indicators, outputs, output indicators, budgets and key risks related to projects. The data should be entered by either the UNDP programme or project manager, with quality assurance conducted by the designated quality assurance officer (see the POPP for more information on the roles and responsibilities in programme and project formulation.)

At the end of the planning process, stakeholders should therefore have as their eighth deliverable—a results framework that may look like the one in Table 12.

Table 13. Sample results framework

National Goal/Priority

“Improved public confidence and involvement in national and local processes of governance” or “more vibrant democratic processes that involve a wider cross-section of citizens”

UNDAF Outcome A1

Wider participation by citizens in national and local elections by 2015

Programme Outcomes

Outcome Indicators, Baselines and Targets

Programme Outputs

Role of Partners

Financial Resources

  1. Electoral administrative policies and systems reformed to ensure freer and fairer elections and to facilitate participation by marginalized groups
  1. Public perception of capacity of electoral management authority to administer free and fair elections (disaggregated by gender, population group, etc.)

Baseline: 40% of public had confidence in electoral management authority as of 2008 (50% men, 30% women, 20% indigenous populations)
Target: 70% of overall population has confidence in electoral management authority by 2016 (75% men, 65% women, 60% indigenous populations)

  1. Advocacy campaign aimed at building consensus on need for electoral law and system reform implemented
  2. Adequate staff recruited and systems implemented in the electoral management authority to administer free and fair elections
  3. Training programme on use of new electoral management technology designed and implemented for staff of electoral management authority

UNDP, Department for International Development (DFID), European Union (EU), US Agency for International Development (USAID) and the World Bank (all working on institutional reform of electoral management authority)

 

  1. Increased participation by women and indigenous populations in national and local electoral processes in five regions by 2016
  1. Percentage of eligible women registered to vote in 5 regions

Baseline: 30% of eligible women registered in the 5 regions as of 2008
Target: 60% registration of eligible women in the 5 regions by 2016

  1. Revised draft legislation on rights of women and indigenous populations to participate in elections prepared

 

UNESCO working on culturally relevant communications programme targeting women and indigenous populations

 

In the UNDAF, all the relevant indicators for the UNDAF outcomes would also be included, along with the outputs of the different UN organizations contributing to those outcomes. Likewise, the national priorities would have their related indicators and outputs in the government’s development strategies. UNDP staff (both programme and operations) should be familiar with these higher level results and performance targets in order to better manage for results in their own programmes and projects.

In UNDP and many other agencies, the information obtained from the planning process is normally used to develop not only the results framework, but also a narrative programme or project document. This document may have requirements that go beyond the issues dealt with in this Handbook. Users of the Handbook should therefore consult with their respective agency policies and procedures manuals for guidance.