In the absence of baseline information and, in most cases, performance indicators, it is difficult to assess project impact in a formal way. Most projects include some form of statement regarding expected results and all are justified as contributions to addressing a major development problem.
The absence of a systematic approach to assessing performance and measuring results is a major deficiency that must be addressed. This major gap in design limits the utility of subsequent evaluation and minimizes the likelihood of learning.
Given this state of affairs, the general approach adopted in this report is to attempt to assess immediate results while also reflecting on broader impact by considering the developmental difference the project in question has made. What follows is a brief, project-by-project review of results and impact, as framed above.
The Alternatives to Slash-and-burn Agriculture (ASB) project has been extremely innovative in that it challenges the concept of biodiversity as a development problem. The project takes a multi-disciplinary approach, bringing in social and economic issues and building links from policy to implementation levels. It stands out as a real effort at SHD advocacy and action and it has brought about significant change on the ground in the countries where it is operating. By proving itself in this way, the project is also stimulating a change in approach to development thinking and practice in both development agencies and national governments. Ironically, there have been difficulties within UNDP in grasping the attempt to capture social and ecological issues simultaneously and in doing so through a participatory approach. The guidelines for GEF projects do not fit easily with such a broad-based, multi-disciplinary, multi-level approach.
The Special Programme for Research and Training in Tropical Diseases project is the ongoing phase of UNDP assistance to the tropical diseases research (TDR) programme of WHO, which began in l975 and which UNDP began to co-sponsor in l976. In its short history, the TDR programme is credited with spearheading the major medical breakthroughs that have taken place in tropical disease research.
In Thailand, through its development of the institutional capability of the Faculty of Tropical Medicine at Mahidol University over the past 20 years, the project has been at the centre of malaria research and the efforts at malaria control which have had remarkable success. Today the Faculty of Tropical Medicine is viewed as a centre of excellence for the whole region. Other countries have benefited from the experience of Thailand and from the vaccines and inoculation campaigns pioneered in the country. In West Africa, the project has provided the foundation for the Onchocerciasis Control Programme, which has virtually eliminated the transmission of river blindness in 11 countries, providing protection to 30,000,000 of the 90,000,000 people at risk.
With regard to new and improved drugs, the TDR programme has registered great progress in the development of drugs, vaccines, diagnostic tests and innovative methods of vector control. In all these initiatives, UNDP funding for the programme has been critical.
Through its careful focus on capacity-building of developing-country institutions and through its training for nationals to undertake their own research as well as the emphasis on collaboration across networks, both North-South and South-South, the project has achieved substantial results over a 20-year period.
The evaluation has raised serious questions about the second phase of the Urban Management Programme (UMP), in which the overall design of the implementation strategy appears to have been ineffective in achieving the objectives or realizing reasonable results. The project’s principal difficulty appears to derive from the lack of fit between the objectives and the kind of activities supported.
However, it is important to take a broader and longer view of the project. It must be recognized that the project has made an important contribution in putting urban management on the map as a development field and in this sense influencing the practice and disposition of the World Bank as well as having a major influence on the Habitat II Conference. Through its research programme and high-quality publications aimed at development practitioners, it has contributed substantially to reinforcing the concept of a multi-sectoral approach to the complex set of problems associated with urban management. In this sense, the project may be said to have had an important developmental impact. It is in its efforts at applications of research and linking policy and implementation levels that the project has been found wanting.
The UNDP/World Bank Water and Sanitation Programme varies in its impact across regions and countries. Generally speaking, the evaluation has found that the project has been important - like the Urban Management Programme - in strengthening the World Bank’s commitment to a multi-disciplinary approach, in this case in the water and sanitation sector, and in making central the concept and practice of community management of the water supply.
More specifically, the project has had a direct impact in helping to shape and reform policy at the country level in Africa. In West Africa, the regional team has had particular success in addressing the restructuring of policy at the country level, notably through consolidation of previously competing ministries. However, the project has had less success in that continent in supporting countries in the development of effective implementation strategies.
By contrast, in Bolivia, there has been no opportunity for the project to influence policy because of political conditions. However, it has done much better in terms of assisting with country implementation strategies in this case as well as elsewhere in Latin America.
As has been noted above, the findings of the evaluation of the Local Initiative Facility for Urban Environment (LIFE) project have been uniformly positive. The project was intended as a pilot effort with the objective of constructing practical models to influence the building of local partnerships to support efforts to tackle urban poverty and improve the condition of the urban environment. The project overall has certainly had a demonstration effect as well as significant local impact. In some cases, it has also influenced the way in which UNDP is designing an element of its country programmes, but the whole approach is new. For UNDP, the project demonstrates the effectiveness of new partnerships in building community-level projects and the positive benefits to be achieved through participatory approaches to project planning and implementation. However, the concept of local-local dialogue is not yet well-understood.
The challenge for the project will be to address this issue through doing better in learning from country experience and deriving and disseminating information regarding best practices. The project has achieved considerable success and may be said to have exceeded anticipated results. It will soon face the necessity of moving beyond the initial phase to broaden and replicate the process and to refine the model on the basis of what has been learned thus far.
The project design does not address this set of issues as directly as it might. Already in Brazil and Pakistan, there are indications that the project is experiencing difficulty in managing activities as new centres of operation begin to open. Long-term impact will depend on the improved sharing and exchange of information and of lessons learned at the interregional level. Close linkages between the LIFE project and the Sustainable Development Networking Programme will also facilitate improved performance in this regard.
The Sustainable Development Networking Programme (SDNP) is one of the most innovative projects in the sample. It involves looking at development through another optic - focusing on the centrality of information in empowerment and building capacity in a very practical way through supporting the establishment of national networks. The project model has been extremely effective and several of the early country partners have now achieved, or are well on the way to achieving, financial self-sufficiency. The project has thus had a demonstration effect and has also produced practical results. Its development impact, over time, may be quite significant although it is early to make such a judgement.
A major limitation of the project is that UNDP has not used the project itself, even within Capacity 2l. Many UNDP country offices have bypassed the network and have invested in their own connections to the Internet at considerable expense.
On the basis of the research undertaken by the evaluation team concerning the project Minimizing Impact of HIV on Development (interregional), there is no hard evidence on the basis of which it is possible to make any statement regarding the impact, results or development model represented by the interregional project as examined in Brazil and Mexico. The project has minimal visibility and minimal presence.
Assistance to the Africa Capacity-building Foundation (ACBF). The project builds on earlier work by the Ford Foundation, IDRC and others in supporting networking and capacity development in the economic policy field. The project is addressing a major development bottleneck: economic policy research and development management. It also represents a direct response to critics of the dependence of African States on foreign expertise and the high cost of technical assistance.
Internal management problems within ACBF have limited the effectiveness of its operations to some degree. Although the project has supported a number of worthy activities, overall it is too early to assess results and impact. The challenges facing the project are immense and the resources inadequate. As noted earlier, it will be essential for the project to focus its objectives on manageable, realistic targets if it wishes to have a real long-term impact.
The Trade Development and Promotion Programme (COMESA) is making a difference and has achieved visible results. Its achievements are largely a result of its successful focusing and definition of the problem. The project has addressed itself to the need to strengthen the primary private-sector vehicle for involvement in trade promotion, namely, the Chambers of Commerce. It has learned from earlier initiatives and, despite many problems, is well focused, well designed and has worked reasonably well. The situation of the private sector in Africa and continuing difficulties in relations between the State and the private sector limit the degree of progress achieved. However, operating in a difficult area, the project has performed effectively.
The National Long-term Perspective Study (NLTPS) is at an early stage, but it has contributed to changing the outlook of the major national stakeholders. In several (although not all) countries where the project is operating, it has opened up a "dialogue culture" among civil society, the private sector, the political leadership and the bureaucracy. The project was conceived as a visioning exercise. As such, the process is more important than the product in terms of its developmental importance. Whether or not it has worked in a specific country is determined by that jurisdiction’s political environment. The project’s results are hard to measure, but it does represent an important example of the way in which UNDP can use its particular position to provide leverage to promote various forms of policy innovation and change.
Water and Sanitation for the Poor in Africa. This regionally funded project, an adjunct to the larger global programme, has helped the global initiative to have greater impact and to target its activities more carefully. It represents an important innovation in linking a regional project to an interregional one, thus facilitating better linkages to country-based activities.
Despite major management difficulties, which had led UNDP to hold back its financial contribution for a period of many months, the project Assistance to the Centre for Environment and Development in the Arab States and Europe (CEDARE) has continued to undertake activities, with the financial gap filled by the Government of Egypt and the Arab Fund. Despite major gaps in its senior management, the project has completed a number of training activities and has developed a niche for itself. It has filled a vacuum in that there has been a long-standing need for a regional environmental training facility. Results achieved have been sufficient for the International Fund for Agricultural Development (IFAD), UNDP and the World Bank, at the country level, to propose working with CEDARE on a project basis. It is too early to expect a new institution of this kind to have broader impact.
As an adjunct to country-based investment programmes by the World Bank and EIB, the Mediterranean Environmental Technical Assistance Programme (METAP) project has been quite successful, principally through its training activities. However, the results achieved thus far are modest, given the size of the financial investment provided by donors. The strong, consistent involvement by UNDP (RBAS and Capacity 21) in the preparation of phase three of METAP has brought about a much tighter focus to project objectives and has produced a much more comprehensive and relevant implementation strategy. The involvement of country teams in preparing national activity lists has also made it more likely that the project will have concrete results and a developmental impact in the sector over a period of years.
The project Institutional Development at the Grass Roots for Poverty Alleviation (South Asia) was intended as a preparatory phase and thus was not designed to produce measurable results. It was intended as a demonstration project and as a means to prepare for a larger, medium-term project. However, there have been problems in making the optimal use of the programme to learn from national experience in building for the longer-term project.
Apparently, there have been some achievements as a result of activities in the project Strengthening Multisectoral and Community Responses to the HIV Epidemic in Asia and the Pacific (Asia regional), notably in South-East Asia and the Pacific, but overall there has been no impact whatsoever in the countries where the project was assessed, namely, India and Pakistan. The project was poorly designed, weakly focused and badly managed. It has had no impact on the national approach to HIV/AIDS, even in the country where the project was based, India. There are a number of mitigating circumstances and the project was halted following a detailed review. There are plans to redesign and relaunch the project.
The project Statistical Institute for Asia and the Pacific (Phase VI) - A Programme in Human Development Indicators, RAS/92/005, was the sixth phase of UNDP assistance to SIAP, which was established in Tokyo in l970 with UNDP funds and which continued as a project until the end of the sixth phase. If one looks at the institution and the effect of UNDP involvement over 25 years, one can say that project funding has had an impact. UNDP funding (in combination with Government of Japan cost-sharing) has contributed to building significant institutional capability. However, this capability is now vested in a Japan-based institution and not in the countries of the region, which are the intended beneficiaries. In Thailand, the project has had a major impact on building the national statistical capability. The situation is the same in many other countries in the region. There are not many projects that can boast of such a singular contribution.
The current phase was intended to address deficiencies in the statistical database of several countries in the region, identified during efforts by UNDP to monitor levels of human development, hence the offer by UNDP to SIAP to fund a programme of human development indicator-related training for countries that most urgently needed it, particularly the least developed countries in the region, the economies in transition, and the island states of the Pacific.
In general terms, the objectives of successive projects have been met. However, it is difficult to evaluate the specific contribution of the sixth phase of support. The PRODOC covers all of SIAP’s programmes and many of these are not supported by UNDP funding. Such a practice poses great difficulties in holding SIAP accountable for the appropriate expenditure of funds and achievement of objectives. It also makes it extremely problematic for UNDP to assess results.
As in the case of the SIAP project, the PRODOC for the ASP-5 Subprogramme on Capacity-building does not distinguish UNDP-supported activities from broader programmes supported by general revenues and other donors. The project was intended to provide guidance and support to a process of institutional reform and, more generally, to support technical improvements to management systems.
Early outputs of the project included the report "Consolidation of ASEAN Institutional Arrangements" along with a management audit. These reports and their recommendations were discussed by four ASEAN bodies before a decision was reached to retain the existing structure.
At the outset of the project and parallel with the adoption of a programme framework for ASP-5 in 1993, the ASEAN Secretariat had committed itself (apparently with the full support of its governing body) to a process of institutional reform and streamlining. The retreat by ASEAN has, for obvious reasons, limited the impact and the value of the project. UNDP can hardly be faulted for the change in its partner's position. As ASEAN's only multilateral dialogue partner, UNDP might reasonably have expected more of its partner in the way of maintaining a commitment.
Despite the setback, according to documents prepared recently for the ASP-5 mid-term review, the project has been effective in a more general way. The ASEAN Secretariat has acted on a number of detailed recommendations and has introduced new management information and financial tracking systems as well as a new approach to the review of staff performance.
The most important dimension of the Project to Support the Development and Enhancement of Democracy, Governance and Participation (DGP) is its timeliness and alacrity in identifying and responding to a significant development gap. The project has been welcomed by countries in the region and has helped to create some space in the countries in transition for dealing with sensitive political issues. It has also begun to create a forum for exposure of one country to developments in others. Despite many initial difficulties and limitations, the absence of a proper implementation strategy or a consultative approach to project planning, the project is of real importance and is valued by respondents in the region.
The project began in difficult circumstances and has been short of funds from the beginning. A number of the deficiencies in project design are being addressed in the planning of a second phase. As with the other projects in the Commonwealth of Independent States and Eastern/Central Europe, the initiation of project investments is relatively recent and it is very early to look at concrete results on the ground.
The "Beijing Express", a modest project, has had very high impact. It is probably without parallel among all the projects considered here in the kind of enthusiasm it has provoked among participants and stakeholders. The project has assisted in publicizing the importance of action in the area of women's issues and addressing the gender dimension. One result has been far greater support from national governments for the efforts by UNDP country programmes to establish gender units and programming oriented towards women. The network effect in building linkages among women leaders in many countries in the region may also have longer-term benefits.
As with the DGP project, it is too early to judge the concrete results of the External Resources Management project. It is well-focused and the combination of training and advisory activities, carefully planned to meet government requirements and particularly timely in responding to an important capacity gap, appears to have been effective. Participation in project activities has been appreciated by trainees as well as their managers.
The project Regional Strategies for the Conservation and Sustainable Management of Natural Resources in the Amazon has had many problems, but in Bolivia, it has contributed to building up an institutional capacity within government and has supported the design of a policy providing for a system of national parks. It has also supported the establishment of a legal framework to protect the environment. These are important contributions, but none of these were among the original objectives of the regional project.
The Poverty Alleviation and Social Development project has had no impact and has achieved no results so far. Approved in 1992, it began operations only in 1994 and at the time of writing, it is being reformulated. A number of weaknesses in project design were identified earlier in this report.
Like several other projects dealing with policy and sensitive political issues, the major contribution of the Human Development and Governance project (see box 8) is the creation of space for the discussion of difficult issues on an intercountry level, linking politics to the development process. These issues are usually kept apart.
This is a very important project. It has high-level support and involvement and has helped to develop ideas and put them into practice. It represents a valuable example of what an intercountry project can contribute in that many of the forms of discussion that took place could probably not have occurred on a single-country basis. The project provides support for governments which might otherwise be hesitant to consider new initiatives and also allows for the bringing to bear of peer pressure in encouraging countries to initiate reform. In addition, it is a very positive example of the way in which UNDP can contribute in a field where other donors, whether bilateral or multilateral, could not.