Forward
Executive Summary

The Regional Bureau for Latin America and the Caribbean (RBLAC) of the United Nations Development Programme (UNDP) financed and supported approximately 3,000 projects from 1988 to 1996. Of these, 200 are concerned specifically with environmental aspects of development or focus primarily on the environment. The present OESP macro-evaluation of 22 projects is the first that focuses on the environmental programmes and projects in the region.

According to the terms of reference, the evaluation was to provide an initial view of the environmental projects as well as guidelines for the examination of specific priority issues in even greater depth and detail at some point in the future.

Its broad objectives were: (a) to assess the extent to which UNDP cooperation has contributed to the achievement of concrete results at the country level in the environmental sector and (b) to identify specific lessons that can be applied in orienting future UNDP investment in this priority area of sustainable human development (SHD).

More specifically, the evaluation was to examine:

• the environmental priorities of the region, as established by the United Nations Conference on Environment and Development (UNCED) and RBLAC, and the nature and value of UNDP support in these priority areas;
• the appropriateness of assigned resources (efficiency);
• the relevance of programmes and projects within the regional environmental agenda and the SHD mandate of UNDP;
• the quality and quantity of outcomes achieved to date (effectiveness);
• the effect of these outcomes on SHD issues (significance);
• the sustainability of the results; and
• the areas where UNDP policies, programmes and projects in this sector need to be reoriented, if any.

Lessons were to be drawn and recommendations were to be made for improving UNDP capacity to provide strategic support to the environment in the region, including consideration of priorities and critical issues in subsectors, geographic concentration, strengthening of management and beneficiary participation.

Twenty-two projects were chosen for analysis according to five main parameters: geographical distribution, type of ecosystem in the project area, environmental priorities of the region as identified by Governments and non-governmental organizations (NGOs), project performance and the age of the project. In view of the average duration of the type of project to be evaluated, the evaluation spanned the period from 1988 to 1996. The field visits were carried out during the summer of 1996.

The projects in the evaluation sample are located in seven countries that represent a wide diversity of natural ecosystems, ecozones and geopolitical features: Argentina (temperate pasture or pampas), Belize and Cuba (coastal ecosystem), Brazil (tropical forest and tropical pasture), Bolivia (Andean ecosystem), Costa Rica (tropical rain-forest), Cuba (island ecosystem) and Guyana (humid tropical ecozone). The selection was the result of several consultations with staff from various UNDP offices, e.g., RBLAC, the Global Environment Facility (GEF) and the Sustainable Energy and Environment Division (SEED) of the Bureau for Policy and Programme Support (BPPS).

The sample included projects in three environmental priority areas: natural resource management, biodiversity and pollution control. Issues such as capacity-building, institutional strengthening, environmental education and the role of women in environmental sustainability were considered in the analysis of the three broad environmental areas.

The criterion of project performance was addressed by evaluating projects that represent all types of implementation experience, including those that have been successful, controversial, interrupted or abandoned. The status of projects varied from those that were completed three years ago to others that are in an advanced phase of execution.

The Office of Evaluation and Strategic Planning (OESP) selected three consultants from the LAC region to undertake the evaluation. Their collective experience included a working knowledge in the fields of environmental management, sociology, engineering, economics and international business. In addition to their professional training, the consultants had extensive field experience in the region in environment and development issues and were familiar with the procedures and development approach of UNDP.

The evaluation team developed the following methodology for its analysis of the projects: (a) gather all available information about a project (project document, interim evaluation reports, publications, news clippings); (b) identify all of the actors relating to that project (including UNDP international and national staff, government counterparts, relevant NGOs, potential beneficiaries of the project, and other people who might otherwise be affected by its implementation; and (c) carry out field visits to observe the project area and interview, individually and collectively, all of the strategic actors to obtain their views on project performance and the impact of the project on the environment.

The evaluators developed guidelines for evaluating the impact of the projects on the environmental priorities of the region. The guidelines, which included numerous questions, consisted of two main sections. The first part focused on how various aspects of the project cycle (including project identification, definition, design, implementation management, participation and coordination among stakeholders, project monitoring and follow-up, ownership, and sustainability) contributed to the achievement of the environmental goals. Part two was directed towards assessing the impact of project activities on the environmental priorities of the LAC region, and particularly the positive environmental gains that have been made.

From their assessment of the 22 projects, the evaluators found the following:

• 59 per cent had a structure that contributed to successful implementation.
• 54 per cent have an effective design that has helped the projects to achieve their objectives.
• In 68 per cent, the roles of the actors were clearly defined.
• 70 per cent have mechanisms for the participation of counterparts and, to a lesser extent, beneficiaries in managing implementation.
• 59 per cent did not have any mechanism for environmental monitoring or any indicators of environmental impact.
• 81 per cent were very flexible during implementation.

• In 50 per cent of the projects, various stakeholders thought that the attitude of UNDP towards those projects has been very positive.
• 68 per cent had resources for contacting groups and institutions that would potentially be affected by and/or benefit from the project.
• 63 per cent included mechanisms for consulting institutions or people who would potentially be affected by the project.
• In 86 per cent, there was political support to implement and follow up the project.
• In only 59 per cent of the projects were there plans to continue the activity initiated by the project after project termination.

• For 50 per cent of the projects, the initial concept was generated by a UNDP country office, UNDP headquarters, a UNDP-managed global programme or a United Nations technical agency; sometimes these initiatives did not coincide with national priorities.
• 27 per cent were identified by Governments according to their own needs.
• 23 per cent were identified by international organizations that have their own agendas or mandates.
• Less than 50 per cent have generated direct participation by grass-roots and community-level stakeholders.
• In 54 per cent, potential beneficiaries participated in the prioritization of project activities.

• Only 41 per cent have been subject to periodic monitoring.
• In the best of cases, the projects were sporadically monitored from the point of view of project implementation but not from that of the environment.

• 72 per cent have been able to manage the conflicts that have arisen during project execution.
• 27 per cent are still facing conflicts during their implementation; 13 per cent of these can be considered as generating a high degree of conflict.

• 72 per cent helped to mobilize technical re-sources for the project.
• 36 per cent helped to mobilize additional financial resources.

• 45 per cent included activities for the communication of results.
• 50 per cent included advocacy activities.

The impact that UNDP projects have had on improving the environmental conditions in Latin America and the Caribbean is directly linked to the performance of the project during the different phases of the project cycle. In other words, the way in which a project has been selected, designed, implemented and monitored affects the potential of that project to improve those conditions.

Conclusion: UNDP has played an important role in project identification. However, environmental projects have been identified according to the availability of funds and/or government interest and not on the basis of an explicit UNDP environmental strategy for the region.

The design phase of a project is a key factor that determines its potential success in improving the environment. For example, when projects are designed by external consultants who are detached from the national context, stakeholders or other strategic actors may not take possession of projects in a timely manner, thus weakening national ownership and sustainability.

Conclusion: Using local consultants contributes significantly to successful project design.

One of the limitations to project efficiency and effectiveness noted by the evaluation team is illustrated by the Low-cost Bamboo Housing Project. When the project was initiated, little research was carried out within UNDP on similar projects that could contribute to a better design of this rather innovative programme. A lack of systematization of data on projects and experience was detected in the country office and in RBLAC. Therefore, projects are unable to profit from the knowledge and past experience of UNDP. In spite of the successful demonstration of bamboo technologies, the project has also made little advancement in transferring its expertise in bamboo cultivation to rural communities. When consulted on this issue, project staff seemed somewhat vague and non-committal.

Conclusion: There is no user-friendly central database from which UNDP staff can retrieve information about UNDP experience on a particular topic.

Additional factors were identified as critical in determining project success.

Regardless of the execution modality or quality of support, all national and NGO counterparts recognized that one of the main comparative advantages of UNDP was its image, including the degree of security and respectability the organization provided to the project and the project personnel. For government institutions, UNDP sponsorship offers autonomy and a degree of protection from the political and financial pressures that are often brought to bear upon public sectors in developing countries; project resources are less likely to be diverted for other uses or mismanaged than when funds are allocated and managed internally.

Although complaints about UNDP bureaucracy are common (particularly when agency execution is involved), it is often considered the lesser of two evils when compared to government administrative procedures that can take months to effect basic purchasing and disbursement operations.

A total of 16 out of 22 projects (70 per cent) have mechanisms for the participation of counterparts and-to a lesser extent-beneficiaries in managing project implementation. The most common mechanism is the steering committee, which advises, assesses, assists with coordination and sometimes exercises approval authority over project work plans, revisions, recruitment and so forth.

The success of the project Patagonian Coastal Zone Management Plan lies in having been able to bring together all of the different stakeholders in the Patagonia coastal area and to coordinate all the diverse activities that the stakeholders were carrying out in favour of the environment. In the short term, this produced concrete outputs and positive outcomes in a rather large area. The creation of awareness and real participation on the part of the public and local government commitment concerning the protection of biodiversity, pollution control, ecological tourism and the need to formulate an integrated plan for the management of the area's natural resources figures among the project's most important achievements.

Conclusion: Participation of stakeholders in a project is a key factor for success.

In theory, the monitoring and evaluation of projects are regular practices in UNDP. Project documents always make reference to tripartite reviews and other regular assessments that must be carried out during the life of the project. However, it was found that in the best of cases, the projects in the sample were sporadically monitored from the point of view of project implementation but not from the perspective of their impact on the environment. A total of 59 per cent of the projects evaluated did not have any mechanism for environmental monitoring or any indicators of environmental impact. In addition, the evaluators found that the monitoring role of UNDP was reactive instead of proactive.

A project succeeds in improving a specific environmental condition when project design takes into consideration existing environmental baseline data, when realistic targets for improving a specific environmental situation are set, and when the necessary scientific, managerial and financial means to achieve the proposed targets are provided.

Conclusion: Baseline data must be developed on the environmental condition or problem that a project will attempt to improve or solve.

Conclusion: UNDP projects must include mechanisms for the monitoring and evaluation of environmental progress.

For the Programme for Sustainable Forestry (Iwokrama Rainforest Programme), project-generated knowledge played a major role in the later development of conservation proposals. In addition, the delimitation of the project's area for forest conservation and sustained use in itself represents an important contribution to the conservation of ecosystems and biodiversity in the country.

Conclusion: UNDP projects are a unique vehicle for the generation of information about the nature and status of natural resources.

Most of the government and NGO representatives interviewed by the team concurred that an essential ingredient for a successful environmental project is the degree of professional education and motivation of UNDP staff in the field of the environment. Not all UNDP staff appeared to be professionally equipped to discuss some environmental issues with national counterparts.

The project document of every UNDP project includes a detailed work plan and timetable. However, very few project managers feel constrained to follow these plans and timetables strictly. There is no accountability for project implementation. If a project is implemented on time according to the timetable in the original project document, this is appreciated, but there does not appear to be enough concern about delays in project implementation.

In many instances, project implementation has depended on UNDP flexibility to be successful. For example, in Brazil, the project Support to the Amazon Working Group of the G7 Pilot Programme has been successful due to the collaboration of the UNDP country office in Brasilia, which has helped various NGOs to prepare their own project proposals for consideration by the Amazon Working Group for funding. In the case of Cuba, the country office has been instrumental in contributing to the organization of international technical workshops within the framework of the regional project Planning and Management of Heavily Contaminated Bay Ecosystems and Coastal Areas of the Wider Caribbean.

Conclusion: The design of a project needs to be flexible and allow for modifications according to changing local conditions.

In Bolivia, the UNDP country office played a decisive role in the creation of the new Secretariat for Indigenous Affairs, Gender and Generation Issues and has provided extensive organizational support to the Under-Secretariat for Indigenous Affairs.

Conclusion: The enthusiastic support of the UNDP country office contributes greatly to project success.

Numerous and dispersed field activities have made logistical demands on the Chaco Ecosystem Programme. Implementation arrangements emphasize the use of short-term consultants. While this has provided flexibility and minimized overhead costs to the Programme, the uneven quality and methodological inconsistency of the consultancy support have often undermined impact. Training activities in particular seem to have been affected.

Conclusion: The logistical arrangements of a project affect its success.

Although the Low-cost Bamboo Housing Project in Costa Rica was a successful undertaking, difficulties arose in understanding the admin-istrative procedures of UNDP. This generated frequent friction between project staff and administrators at the UNDP country office. The administrative needs of environmental projects can surpass field office capacities, particularly when they require a constant stream of small, scattered disbursements to individuals or informal community organizations at short notice. The bamboo project as well as the GEF Small Grants Programme (SGP) and the Support to the National Indigenous Programme, both in Bolivia, are familiar with the difficulties involved in servicing dispersed, micro project activities that are susceptible to external factors such as distance, climate, agricultural calendars or community labour cycles.

Conclusion: To promote SHD, UNDP must (a) improve its performance in contacting groups/institutions that potentially could benefit from/be affected by its projects; (b) increase the participation of potential beneficiaries in the prioritization of project activities; and (c) increase its ability to resolve the conflicts generated by its projects.

It is somewhat difficult to assess the impact that UNDP-financed/coordinated natural resource and biodiversity projects have had on the sustainable management of natural resources for two main reasons. First, environmental baseline data and in-depth analysis of the present status of the natural resources of the region are lacking. Thus, it is not easy to ascertain whether a UNDP project on the sustainable management of tropical forests, for example, has or has not improved the general situation of the rainforests of the region since the status of the rainforests was not accurately known before the start of the project. Second, the real impact of a project on natural resource management frequently becomes apparent only after a long period.

Despite these limitations, and after examining the nine natural resource projects included in the sample, the evaluation mission found the following:

• 60 per cent of the projects contributed to the development of baseline studies and data on natural resources.
• 46 per cent contributed to the establishment or enrichment of data banks where information on natural resources collected by projects is deposited.
• 23 per cent developed systems for the sustainable management of natural resources.

• 30 per cent developed economic incentives to protect and conserve natural resources.
• Only 50 per cent paid any specific attention to the generation of economic gains from the sustainable management of natural resources.
• 70 per cent trained personnel to empower them to promote the sustainable management of natural resources.

• 53 per cent contributed to the development of educational activities (e.g., development of case studies and educational materials for schools) relating to the sustainable management of natural resources.

Some UNDP-sponsored projects have helped to introduce new technologies that are of paramount importance in conserving and protecting biodiversity and natural resources, e.g., the changes in fishing technology introduced under the Patagonian Coastal Zone Management Plan.

Conclusion: By introducing environmentally sound technologies, UNDP has contributed significantly to the promotion of the sustainable management of natural resources.

Since the preparatory phase of the Low-cost Bamboo Housing Project in Costa Rica, for example, 737 low-income housing units have been built in rural communities and approximately 30 additional units are presently under construction. Of this total, 90 were built entirely by local beneficiaries through community self-help initiatives; the remainder have combined self-help and recruited labour. Each constructed unit has provided tem-porary employment for eight persons. Building methods are environmentally sound, simple and labour-intensive; bamboo panels are plastered with cement to give the appearance of a standard home. Self-help housing recipients are able to choose from four different housing models, designed in consultation with communities; when financing is obtained through BANHVI (the housing bank), beneficiaries can choose from two models.

UNDP-sponsored projects have succeeded in interesting people in the conservation of local biodiversity resources when the projects have highlighted the potential economic benefits that the local population can derive from protecting that biodiversity, managing it in a sustainable manner. For example, the Chaco Ecosystems Programme has contributed to the identification and establishment of conservation measures to exploit, in a sustainable fashion, the shad population in the Pilcomayo River. "Responsible Tourism" was the motto of the ecological tourism programme developed by the project Patagonian Coastal Zone Management Plan.

Conclusion: By linking economic opportunities with resource management, UNDP has contributed to the promotion of SHD.

Conclusion: The training of local people in environmental aspects of natural resources is an important factor in ensuring the protection and sustainable management of those resources.

One of the criteria the evaluators used to measure the impact of UNDP projects on the environ-ment was the sustainability of project activities. The team sought answers to such questions as: "What happens after a project ends?" "Will the Government, institutions and other stakeholders continue the initiatives that the project started?" The results of its inquiries are described below.

Conclusion: Sustainability cannot be reliably assessed when provisions for ex-post monitoring or follow-up support are lacking.

Conclusion: The sustainability of project activities is directly related to the degree of participation of various stakeholders in project design and implementation. The greater their participation, the better the chances for sustainability.

Conclusion: A number of natural resource and biodiversity projects have developed a high level of dependency on UNDP. Thus, the initiatives begun by those projects will be sustained only if further funding is provided.

Conclusion: The sustainability of project activities depends on the political will of the authorities to continue those activities.

Conclusion: Sustainability of project activities also depends on a project's economic performance.Many women in Latin American countries work on the land and use resources (water, coal and other energy inputs) to provide food and shelter for their families. Unfortunately, none of the projects analysed by the evaluation team showed any specific, explicit effort to enhance the participation of women in the protection and conservation of natural resources and local biodiversity.

Conclusion: The participation of women in the protection and sustainable use of their local environment is of paramount importance to the sustainability of the activities and results of environmental projects.

The analysis of the five projects with a focus on reducing and preventing pollution shows that UNDP has contributed to pollution reduction in some areas of Latin America and the Caribbean. The main findings of the evaluation team are:

• General
• 50 per cent have accomplished their objectives as indicated in the project document.
• All of the projects focused on planning for pollution reduction, but only one had a specific activity to reduce pollution.

• 80 per cent of the projects contributed to the development of an inventory of pollutants and pollution sources.
• 80 per cent systematized and published the scientific data gathered during project implementation.
• Although all of the projects intended to reduce pollution in their geographical area of influence, none established specific targets for reducing the pollution problem they were tackling.

• None of the projects related their activities/results to the potential/existing economic gains that would derive from reducing pollution in the project's area of influence.

• All of the projects influenced the development of local environmental policies and norms.
• Only 50 per cent contributed to the enactment of new environmental policies.

• The activities of only one project have shown a high degree of sustainability.

Conclusion: Knowledge of and access to baseline data on existing pollutants and sources of pollution are essential determinants of a project's success in reducing pollution.

Conclusion: UNDP needs to develop indicators of environmental performance in order to promote projects aimed at reducing pollution.

Conclusion: The participation of the local population is a key factor in reducing pollution.

Conclusion: Encouraging the participation and collaboration of institutions involved in the monitoring of pollution increases the likelihood and extent of project success.

To spend resources only on reducing pollution is an inefficient allocation of those resources. UNDP pollution-oriented projects have a greater impact when the economic benefits derived from pollution reduction are clearly demonstrated.

Conclusion: UNDP projects to reduce pollution need to identify the potential economic gains to be derived from that reduction.

Twelve of the 22 evaluated projects refer to capacity-building and some form of institutional strengthening. However, many of them still have a great deal of work to do with regard to obtaining information on two topic: the institutions themselves and the extent of the improvement in the management of the institutions where projects were developed. On the other hand, many of the projects made contributions to the education and training of the staff of the institutions influenced by the projects.

Four projects focused on institutional strengthening and capacity-building. The main findings from the assessment of those projects are:

• 75 per cent contributed to the formulation of national/local environmental policies, laws or environmental standards.
• 50 per cent helped to improve environmental education programmes in the country/region.
• 75 per cent contributed to the development of databases about the environmental conditions of the country/region where the project was implemented.
• All of the projects developed networks with NGOs and representatives of civil society involved with the environment.
• None of the projects contributed to further strengthening of links with the private sector.

The evaluation mission reached the following conclusions.

Conclusion: UNDP has contributed significantly to enhancing the skills of professionals as well as other people in local communities to improve their capacity to manage the environment.

Conclusion: UNDP has also made an important contribution to the strengthening of national and local environmental institutions.

After assessing the ways in which UNDP has designed, developed and implemented its environmental projects, the evaluation team made the following recommendations.

• To derive the greatest benefit from its resources, UNDP should develop an explicit, long-term regional environmental development strategy that will enable UNDP to maximize its contribution in areas that are regional priorities.
• The strategy should include criteria for project identification and selection, including consideration of the potential impact that the identi-fication, selection and implementation of the project could have on the environment of a particular region.
• UNDP should establish specific targets to improve specific environmental problems, e.g., to reduce the rate of deforestation of Latin America and the Caribbean by one per cent by the year 2003.
• UNDP should make the use of the existing UNDP Handbook and Guidelines for Environmental Management and Sustainable Development mandatory for all UNDP staff in order to ensure that the environmental dimension is explicitly included in all UNDP-sponsored projects and activities.

• All environmental projects should be identified according to national and regional environmental priorities.

• All UNDP-sponsored environmental projects should develop baseline data about the environmental situation or problem that the projects will attempt to improve or solve.

• UNDP should develop a user-friendly data bank that would include the experience gained from its own environmental projects.
• The environmental data bank should contain names of people and institutions that have participated in projects, a description of project outcomes and results and a brief summary of lessons learned.
• UNDP staff who are preparing new environmental projects should always use the data bank in order to benefit from the valuable experience gained by the organization throughout the years.

• Local environmental consultants should be used whenever possible throughout the entire project cycle since they contribute significantly to successful project design and help to strengthen national ownership and sustainability.

• The situation or problem that a project is attempting to improve or solve should be clearly defined and the project objectives should be unambiguously and concisely stated.

• At the project design stage, UNDP staff should identify and consult with the potential beneficiaries or people who might otherwise potentially be affected by the implementation of a particular environmental project.
• The capacities of local stakeholders and beneficiaries to contribute to project implementation should be assessed.
• People involved in project design should have a thorough knowledge of the local project environment.
• Project designs should be flexible and allow for modifications according to changing local conditions.

• · UNDP projects should include:
  
  • clear indicators of environmental performance (how results/success can be measured);

  • mechanisms for the monitoring of environmental progress (monitoring will, of course, be possible only with the development of baseline data, performance indicators and an explicit strategy that indicates what the project is attempting to achieve);

  • mid-term and final evaluations where the results of the project are assessed;

  • a means of assessing project results in terms of the fulfilment of the needs of the national government counterparts.

• Staff responsibilities for implementing project activities should be clearly stated along with a schedule for such activities and for project deliverables.
• The roles of participating national counterparts should be clearly described.
• Projects should include a detailed budget that is adequate for the financing of each of the activities envisaged under the project.

• Projects should share information with all of the various actors of the project (UNDP staff, national counterparts, beneficiaries and other people potentially affected).
• The results of all UNDP-sponsored environmental projects should be shared with host governments and local institutions.

• UNDP should continue its efforts to enhance systematically the environmental management education and training of all of its personnel (both at headquarters and in the field).
• Training should include some general capacity-building activities as well as specific training in identifying the environmental performance of projects.

• UNDP should make a greater effort to carry out capacity-building activities in the field of the environment to provide people with the knowledge and tools that will improve their ability to protect, manage and conserve their environment.

• UNDP should make a systematic effort to encourage explicitly the participation of women in environment-related projects as agents of conservation and promoters of SHD. This means that each environmental project should include capacity-building, training and environmental monitoring activities specifically targeted to women.