Document Library

Key Text A Practitioner's Guide to Evaluating the Impacts of Labor Market Programs

Author: Emla Fitzsimons and Marcos Vera-Hernández
Date: 2009
Size: 12 pages (241 kB)

Access full text: available online

Summary

How can the impact of labour market programmes in developing countries be credibly evaluated? This note outlines the main issues that need to be considered when planning an impact evaluation. It also covers the techniques used to estimate impacts. Knowledge of evaluation techniques is important even at the programme planning stage; the types of data that need to be collected will influence the optimal design of a programme pilot.

An impact evaluation of a labour market programme is a quantitative analysis that estimates the difference the programme makes. Evaluations specify one or more outcome variables of interest (earnings or employment for example) and estimate the difference a programme makes to these variables. Evaluations have to establish the counterfactual: what would have happened without the programme. They also need to overcome any bias, including selection bias in comparing participants and non-participants.

Issues to consider when conducting evaluations include: the choice between individual-level and cluster-level evaluation; the type of impact being evaluated; and data requirements.

An individual-level evaluation uses data from participants and non-participants living in the same communities. So within each community, participants and non-participants must be observed. A cluster-level analysis must distinguish between treatment communities and control communities.
There are two types of impacts that an evaluation can estimate: the impact of the treatment itself, and the impact of being offered the treatment (regardless of whether it was received or not). The latter is called the intention to treat impact.
There is no data checklist that applies to all evaluation techniques, but data are required on the outcomes of interest and on background characteristics, or 'covariates'.

Different techniques can be applied to data to estimate the counterfactual, and thus the impact of a programme:

Randomisation: In its pure form, this allocates participants and non-participants randomly. In practice, political considerations can interfere. There are variants on the randomisation technique to deal with this.
Matching: This estimates the counterfactual using data on non-participants. It gives more weight to non-participants who are more similar to participants, as measured by observable background characteristics.
Difference-in-Differences: This recognises that participants and non-participants would have different average outcomes even in the absence of the programme. The assumption is that differences in outcome due to differences in unobserved background characteristics do not change over time.
Instrumental Variables: This technique assumes there is one or more instrumental variable that only affects outcomes through affecting programme participation.
Regression Discontinuity: This takes account of the fact that eligibility to participate in a programme is sometimes determined on the basis of a threshold on an index.

Programmes should be designed to allow for credible evaluation. A pilot stage is important to test whether the programme works and try different variants to see which works best.

The evaluation techniques outlined here are useful to know before the programme is piloted. They can strongly influence the types of data that need to be collected and the optimal design of the pilot.
Randomisation is widely considered the gold standard of evaluation. But not all programmes can be randomised and other techniques can be used in these situations.
Deciding which labour market programmes to implement with limited resources can only be done with careful planning and rigorous evaluation.

Access full text: available online

Source: Fitzsimons, E. and Vera-Hernández, M., 2009, 'A Practitioner's Guide to Evaluating the Impacts of Labor Market Programs', World Bank Employment Policy Primer No.12, World Bank, Washington, D.C.
Organisation: World Bank, http://www.worldbank.org/