Monitoring and evaluation

Managing M&E

Page contents

• Steps in Planning and Design
• Ensuring Evaluation Quality
• Identifying Threats to the Validity of Evaluation Findings 
• Evaluating Multi-donor Programmes
• The Politics of Evaluation
• Promoting the use of Evaluation Findings
• Strengthening National M&E Capacity

Badly designed and managed evaluations can do more harm than good - misleading results can undermine the effective channelling of resources for poverty reduction.

Establishing international standards for methodological rigour, ethical practice and efficient management processes in monitoring and evaluation is an ongoing challenge. Key concerns are how aid agencies should oversee evaluations outsourced to consultants, how to build country ownership of M&E processes where there are significant capacity constraints or limited buy-in, and how to co-ordinate evaluations of joint donor programmes effectively.

Steps in Planning and Design

Monitoring and evaluation activities are usually broken down into stages of planning, implementation, analysis, dissemination and use. 

Kusek, J., and Rist, R., 2004, ‘Ten Steps to a Results-based Monitoring and Evaluation System’, World Bank, Washington
Governments and organisations face increasing internal and external pressures to demonstrate accountability, transparency and results. Results-based monitoring and evaluation (M&E) systems are a powerful public management tool to achieve these objectives. This handbook from the World Bank presents a ten-step model that provides extensive detail on building, maintaining and sustaining a results-based M&E system.      
Access full text: available online

Aid agencies largely work to their own internal requirements for reviewing, reporting on and evaluating the inputs, process and results of their activities, producing internal guidance notes that describe the practical steps involved. 

DFID, 2005, ‘Guidance on Evaluation and Review for DFID Staff’, Evaluation Department, DFID, London
Good evaluation practice depends on a solid partnership between those commissioning and managing evaluation studies and the consultants undertaking the work and producing reports. This guide from DFID aims to improve the quality of decentralised evaluation. It outlines the steps for designing, managing, reporting on and responding to an evaluation. 
Access full text: available online

United Nations Development Programme Evaluation Office, 2002, 'Handbook on Monitoring and Evaluating for Results', UNDP, New York
Since 1999, the United Nations Development Programme (UNDP) has placed greater emphasis on results in its work to eliminate poverty. That shift has led to new demands on Monitoring and Evaluation (M&E) in country offices and programme units. This handbook outlines an M&E framework for use by UNDP staff and partners that promotes learning and performance measurement.
Access full text: available online

Ensuring Evaluation Quality

Ensuring the quality and integrity of evaluation design is vital for reaching accurate and reliable conclusions about what works and what doesn’t work in development. International standards emphasise the need for impartiality, appropriately skilled experts conducting the evaluation, participation, country ownership and timeliness (evaluations should be appropriately timed to influence policymaking).

The OECD DAC Network on Development Evaluation’s principles for evaluation of development assistance are widely cited. 

OECD-DAC, 1991, ‘Principles for Evaluation of Development Assistance’, OECD-DAC, Paris
Aid evaluation plays an essential role in efforts to enhance the quality of development cooperation. This paper from the OECD's Development Assistance Committee presents a set of principles on the most important requirements of the evaluation process. Development assistance is a cooperative partnership between donors and recipients. Both must take an interest in evaluation to improve the use of resources through learning and to ensure accountability to political authorities and the public.   
Access full text: available online

There is also a need to ensure that evaluations are conducted ethically, meaning in a culturally sensitive manner that protects the anonymity and confidentiality of individual informants.

United Nations Evaluation Group, 2005, ‘Standards for Evaluation in the UN System’, UNEG, New York
An effective evaluation process is an integral part of any project. But what are the key elements of a successful and sustainable evaluation approach? This document produced by the United Nations Evaluation Group (UNEG) offers solid guidelines for evaluation planning, design, implementation and reporting. Fundamental requirements include: institution-wide support, clearly-defined and transparent responsibilities, appropriately qualified staff, and a constant commitment to the harmonisation and updating of methods used.
Access full text: available online

Identifying Threats to the Validity of Evaluation Findings

Significant criticism has been levelled against the development community for failing to adopt methodologically sound approaches to evaluating their activities. These include weak analysis of qualitative data and not paying enough attention to mapping the causal chain from inputs to impacts. 

White, H., 2005, ‘Challenges in Evaluating Development Effectiveness’, IDS Working Paper 242, Institute of Development Studies, Brighton
Evaluation has a crucial role to play in today?s results-based culture and in the context of the Millennium Development Goals (MDGs). How then, can the quality of evaluation be improved? This working paper from IDS (Institute of Development Studies) argues that there has been inadequate investment in methodology, often resulting in low quality evaluation outputs. It discusses techniques in three areas of contemporary relevance: measuring agency performance; evaluation methods at the project level; and sustainability analysis.  
Access full text: available online

The validity and usefulness of an evaluation are determined, amongst other things, by its statistical validity, use/action orientation, transferability and fittingness.  

Bamberger, M., Rugh, J., and Mabry, L., 2006, ‘Strengthening the Evaluation Design and the Validity of the Conclusions’ Chapter 7 in Realworld Evaluation: Working Under Budget, Time, Data and Political Constraints, Sage Publications
How can threats to the validity of evaluations be identified and addressed?  This chapter from Realworld Evaluation:  Working Under Budget, Time, Data and Political Constraints outlines some of the most common threats to the validity of both quantitative (QUANT) and qualitative (QUAL) evaluation designs.  It offers recommendations on how and when corrective measures can be taken to protect validity.
Access full text: via document delivery. Please see document summary.
 

Bamberger, M., Rugh, J., and Mabry, L., 2006, ‘Integrated Checklist for Assessing the Adequacy and Validity of Quantitative, Qualitative, and Mixed-Method Design’, Appendix 1 in Realworld Evaluation: Working Under Budget, Time, Data and Political Constraints, Sage Publications 

Whilst many of these methodological issues are recognised, overcoming them is difficult in practice, especially where there are limited funds designated for evaluation.

Green, A., and Kohl, R., 2007, ‘Challenges in Evaluating Democracy Assistance: Perspectives from the Donor Side’, Democratization, vol 14, no. 1, pp. 151-165
Why is there a lack of credible research into the impact of democracy assistance? What are the obstacles to conducting such research? This article from Democratization shares insights from a donor-sponsored workshop on the challenges facing the evaluation of democracy and governance (DG) programming and assistance. It argues that the lack of credible research is partly due to a fundamental difference in orientation between the retrospective approach of academics and the prospective approach of donor agencies.
Access full text: available online

Evaluating Multi-donor Programmes

The Paris Declaration commits donors to cooperation and harmonization in all stages of the development cycle. Joint evaluations are necessary where multiple agencies are involved in a chain of interventions to pursue similar outcomes, or to understand the combined effects of all interventions across a particular sector.

Joint evaluations present opportunities for donors to pool their technical and financial resources into more rigorous, in depth and longer-term evaluations and in turn reduce the multiple information demands on governments and stakeholders. Neverthless, they require reconciling the often divergent mandates and preferred evaluation approaches of different agencies.

OECD-DAC, 2006, ‘Guidance for Managing Joint Evaluations’, DAC Evaluation Series
Joint evaluations have become central to development practice in recent years. Collective assessment of agencies? combined work minimises transaction costs for developing country partners and addresses the large aid-giving role of joined-up modalities such as basket funds and joint assistance strategies. This booklet produced by the Development Assistance Committee of the Organisation for Economic Cooperation and Development (OECD-DAC) provides practical guidance for making joint evaluations efficient, educational and collaborative.  
Access full text: available online

OECD-DAC Network on Development Evaluation, 2007, ‘Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative Principles and Standards’, OECD, Paris
Global and Regional Partnership Programmes (GRPPs) are an increasingly important modality for channeling and delivering development assistance. This World Bank Independent Evaluation Group (IEG) Sourcebook, prepared under the auspices of the OECD/DAC Network on Development Evaluation, is designed to address the growing need for consensus principles and standards for evaluating GRPPs. It comprehensively presents, synthesises, applies and elaborates on existing principles and standards, aiming to improve evaluation independence and quality. As a result, GRPPs should become more relevant and effective.
Access full text: available online

The Politics of Evaluation

Evaluations are more than a technical process. They have the capacity to determine access to resources and the funding fate of programmes. It is inevitable therefore that they will be subject to pressures from different stakeholders to produce favourable assessments or to avoid addressing sensitive issues.

Bamberger, M., Rugh, J., and Mabry, L., 2006, ‘Reconciling Different Priorities and Perspectives: Addressing Political Influences’, Chapter 6 in Realworld Evaluation: Working Under Budget, Time, Data and Political Constraints, Sage Publications
No evaluation can ever be value free and completely objective. Decisions as to what to study, which methods to use and whose criteria define programme success all involve human judgement. This chapter from RealWorld Evaluation: Working Under Budget, Time, Data and Political Constraints, discusses how political factors affect evaluation. It provides a detailed analysis of possible pressures and constraints in evaluation design, implementation, dissemination and use.
Access full text: via document delivery. Please see document summary.

Promoting the Use of Evaluation Findings

M&E should ultimately result in improved policy and practice. Yet the findings and recommendations of evaluations are frequently underutilized. In order for evaluations to be influential, it is important to consider how to integrate them into the policymaking cycle, the political incentives to take up findings, and how the report is presented and understood by different stakeholders.

World Bank Operations Evaluation Department, 2004, 'Influential Evaluations: Evaluations that Improved Performance and Impacts of Development Programs', World Bank, Washington, D.C.
Evaluations can be a cost-effective way of improving the performance and impact of development activities. However, they must be conducted at the right time, focus on key issues and present results in an accessible format. This report from the World Bank Operations Evaluation Department presents eight examples of evaluations that have had an important impact, and summarises lessons learned.
Access full text: available online

The relative influence of evaluations also depends on whether there are sufficient institutional arrangements to support the transformation of policy lessons into policy actions.

Gordillo, G., and Andersson, K., 2004, ‘ From Policy Lessons to Policy Actions: Motivation to Take Evaluation Seriously’, Public Administration and Development, Volume 24, pp. 304-320
Whilst recent political reforms have sometimes led to modifications in countries? national policies, the link between policy evaluation and policy actions is often weak. So why do so many governments take policy evaluation so lightly? This article from the Food and Agriculture Organisation (FAO) and the Center for the Study of Institutions, Population, and Environmental Change (CIPEC) at Indiana University analyses the institutional aspects of creating effective systems for monitoring and evaluations (M&E) in government-led rural development efforts in Bolivia and Brazil.  
Access full text: available online

Strengthening National M&E Capacity

Evaluations of development programmes have historically been driven and designed by donors - primarily to satisfy their own accountability needs. However, it is increasingly recognised that both monitoring and evaluation should be a country-led process, not least because country ownership is a major factor in determining whether evaluation findings are then used in a national context.

Technical barriers to country-led evaluations centre around lack of human and financial resources, but M&E is also a highly political issue and the incentives, or lack of incentives, for evaluations to be conducted (e.g. fear of aid being withdrawn as a result of negative evaluation results) also need to be considered.

There is evidence that capacity development programmes work best when adapted to the local governance structure, professional capacity and evaluation culture.

Schiavo-Campo, S., 2005, ‘Building Country Capacity for Monitoring and Evaluation in the Public Sector: Selected Lessons of International Experience’, World Bank  Evaluation Capacity Development Working Paper no. 13, World Bank, Washington
The Evalution Capacity Development (ECD) unit of World Bank's Operations Evaluation Department is designed to help countries strengthen their monitoring and evaluation (M&E) capacity. The unit targets 'high-intensity' support to Uganda and Egypt and various other types of support to an additional 32 countries. This paper from the World Bank collates some of the main lessons learned from ECD activities and outlines the major issues which need to be addressed. 
Access full text: available online

New additions