Monitoring and evaluation

Impact evaluation

Page contents

• Attribution and the Counterfactual: The Case for More and Better Impact Evaluation
• Randomized Control Trials: The Gold Standard?
• Adapting to Time, Budget and Data Constraints
• Mixed-method Designs
• Toolkits

The recent emphasis on accountability and results-based management has stimulated interest in evaluating not just the process, outputs and outcomes of development programmes, but also their impact (ultimate effect) on people’s lives. Impact evaluations go beyond documenting change to assess the effects of interventions on individual households, institutions, and the environment, relative to what would have happened without them – thereby establishing the counterfactual.

This rigorous approach to evaluation is increasingly advocated as the only reliable way to develop an evidence base of what works and what doesn’t in development. Nevertheless, impact evaluations remain relatively rare, and in practice evaluation methodology often has to be adapted to technical, political and capacity constraints.

Attribution and the Counterfactual: The Case for More and Better Impact Evaluation

Development interventions are not conducted in a vacuum, and it is extremely difficult to determine the extent to which change (positive or negative) can be attributed to the intervention, rather than to external events (such as economic, demographic, or policy changes), or interventions by other agencies.

Impact evaluations attempt to attribute change to a specific programme or policy and establish what would have happened without the intervention (the counterfactual) by using scientific, sometimes experimental, methodologies such as randomized control trials or comparison groups.

The Evaluation Gap Working Group at the Centre for Global Development is the foremost advocate for the use of such rigorous evaluation methodologies to address what it views as a critical gap in evidence about the real impact of development programmes.

Center for Global Development, 2006, ‘When Will We Ever Learn? Improving Lives Through Impact Evaluation’, Evaluation Gap Working Group, Centre for Global Development, New York
Despite decades of investment in social development programmes, we still know relatively little about their net impact. So why are rigorous social development impact evaluations relatively rare? This paper from the Center for Global Development (CGD) aims to address this question and provide recommendations for more and better evidence for policymaking and programme planning. A new, collective approach is needed, in which developing country governments, bilateral and multilateral development agencies, foundations and NGOs work together to define an agenda of enduring questions and fund the design and implementation of rigorous impact evaluations in key sectors.
Access full text: available online

Randomized Control Trials: The Gold Standard?

Randomized Control Trials (RCTs) are often referred to as the ‘gold standard’ of impact evaluation, but whether or not they are always feasible, appropriate and rigorous is the subject of some debate.

Duflo, E., and Kremer, M., 2003, ‘Use of Randomization in the Evaluation of Development Effectiveness’, Paper prepared for the World Bank Operations Evaluation Department (OED) Conference on Evaluation and Development Effectiveness 15-16 July, 2003, Massachusetts Institute of Technology , Cambridge, Mass 
Just as randomised pharmaceutical trials revolutionised medicine in the 20th Century, randomised evaluations  could revolutionise social policy in the 21st. This paper, prepared for a 2003 World Bank Operations Evaluation Department (OED) conference, draws on evaluations of educational programmes. It argues that there is an imbalance in evaluation methodology and recommends greater use of randomised evaluations. As credible impact evaluations, these could offer valuable guidance to international organisations, governments, donors and NGOs in the search for successful programmes.
Access full text: available online

Case Study: Progresa in Mexico

The Progresa case is considered one of the most successful examples of the application of a randomized control trial in a development context.

Attanasio, O., Meghir, C., and Santiago, A., 2005 ‘Education Choices in Mexico: Using a Structural Model and a Randomized Experiment to Evaluate Progresa’, Institute for Fiscal Studies (IFS), London
What impact have monetary incentives had on education choices in rural Mexico? How can the design of educational interventions aimed at improving educational participation be improved? This paper from the Institute for Fiscal Studies (IFS) analyses the education component of the Mexican governments welfare programme, Progresa, which aims to reduce rural poverty. It argues that increasing the grant for secondary school children while eliminating it at the primary age would strengthen Progresas impact.     
Access full text: available online

For further case studies of randomized control trials, see: http://www.povertyactionlab.com

 

Adapting to Time, Budget and Data Constraints

Ideological positions can obscure the issue of what methodologies are actually feasible. Scientific approaches can be costly, time consuming, and therefore unrealistic. Many organisations don’t have the resources to carry out the ideal evaluation, and an M&E framework needs to be designed with organisational capacity, human and financial resources and political context in mind.

It is important to understand the minimum methodological requirements for evaluation rigour in cases where it is not possible to use strong evaluation designs.

Bamberger, M., 2006, ‘Conducting Quality Impact Evaluations Under Budget, Time and Data Constraints’, World Bank, Washington
How do cost, time and data constraints affect the validity of evaluation approaches and conclusions ? What are acceptable compromises and what are the minimum methodological requirements for a study to be considered a quality impact evaluation? This booklet from the World Bank provides advice for conducting impact evaluations and selecting  the most rigorous methods available within the constraints faced. It provides suggestions for reducing costs and increasing rigour and clarifies the nature of trade-offs between evaluation rigour and budgets, time and data.  
Access full text: available online

Bamberger, M., Rugh, J., and Mabry, L., 2006, ‘Bringing It All Together: Applying Realworld Evaluation Approaches to Each Stage of the Evaluation Process’ Chapter 16 in Realworld Evaluation: Working Under Budget, Time, Data and Political Constraints, Sage Publications
How can the principles of optimal evaluation design be applied under real world conditions with budget, time, data and political constraints? This paper, adapted from chapter 16 of RealWorld Evaluation: Working under Budget, Time, Data and Political Constraints provides an overview of the RealWorld Evaluation (RWE) approach. It addresses constraints through practical suggestions applicable to both developing and developed world research. Understanding the aims and purpose of the evaluation, as well as the local context, is critical.
Access full text: available online

Evaluation designs need to adapt to local realities, and experience demonstrates that no single methodology is applicable in all cases.

White, H., 2006, ‘Impact Evaluation: The Experience of the Independent Evaluation Group of the World Bank’, World Bank, Washington
Aid spending is increasingly dependent on proof that interventions are contributing to the attainment of Millennium Development Goals (MDGs). Yet there is still debate over the definition of impact evaluation and how it should be carried out. This paper draws on the experience of the Independent Evaluation Group (IEG) of the World Bank. It defines impact evaluation as a ‘counterfactual analysis of the impact of an intervention on final welfare outcomes’ and recommends a theory-based approach. Two sources of bias are highlighted: contamination and self-selection bias.
Access full text: available online

Mixed-Method Designs

The debate on whether randomized control trials are the ‘gold standard’ of impact evaluation is largely based on the assumption that only quantitative evaluation designs are rigorous. In reality, effective evaluations may require both qualitative and quanititative analysis of results. Combining both types of analysis (using the ‘mixed-method approach’) can overcome the weaknesses inherent in either method.

Bamberger, M., Rugh, J., and Mabry, L., 2006, ‘Mixed-Method Evaluation’ Chapter 13 in Realworld Evaluation: Working Under Budget, Time, Data and Political Constraints, Sage Publications
Quantitative and qualitative methods of research each have strengths and weaknesses when applied in isolation. However, combining the two approaches through mixed-method evaluation is gaining wider acceptance among social science researchers as a way of conducting more comprehensive and robust analysis. This chapter from RealWorld Evaluation: Working Under Budget, Time, Data and Political Constraints discusses the most appropriate contexts and strategies for using a mixed-method approach. It argues that mixed-method evaluation is a flexible and practical technique which can be used at any stage of an evaluation. Nevertheless, a fully integrated approach requires extensive planning and deliberation to ensure that the most appropriate combination of methods is chosen and successfully implemented.
Full document: available via document delivery. Please see document summary.

Toolkits

Baker, J., 2000, ‘Evaluating the Impact of Development Projects on Poverty: A Handbook for Practitioners’, World Bank, Washington
There is broad evidence that developmental assistance benefits the poor, but how can we tell if specific projects are working? Have resources been spent effectively? What would have happened without intervention? This comprehensive handbook from Directions in Development seeks to provide tools for evaluating project impact. It advises that effective evaluations require financial and political support, early and careful planning, participation of stakeholders, a mix of methodologies and communication between team members.
Access full text: available online

The World Bank, 2006, ‘Impact Evaluation and the Project Cycle’, PREM Poverty Reduction Group, World Bank, Washington
The goal of an impact evaluation (IE) is to attribute impacts to a project using a comparison group to measure what would have happened to the project beneficiaries had it not taken place. The process of identifying this group, collecting the required data and conducting the relevant analysis requires careful planning. This paper from the World Bank provides practical guidance on designing and executing IEs. It includes some illustrative costs and ideas for increasing government buy-in to the process.
Access full text: available online

New additions