Broadening the Range of Designs and Methods for Impact Evaluations
Author: Elliot Stern, Nicoletta Stame, John Mayne, Kim Forss, Rick Davies, and Barbara Befani
Access full text: available online
Investment in Impact Evaluation (IE) has focused on a narrow range of mainly experimental and statistical methods and designs. DFID has found, however, that these are only applicable to a small proportion of its programmes. This report presents the findings of a study that considered existing IE practice, reviewed methodological literatures and assessed how state-of-the art evaluation designs and methods might be applied to contemporary development programmes.
The report notes that most development interventions 'work' as part of a causal package in combination with other factors. It finds that some of the most potentially useful approaches to causal inference are not generally known or applied in the evaluation of international development and aid. A broader range of IE designs and methods – including theory based, case-based and participatory approaches – could, if suitably developed and applied, extend IE to programmes and contexts where it is currently difficult. The report's main conclusions are outlined below.
- IE can best be understood as a form of causal analysis that links an intervention with effects for beneficiaries. Explaining cause and effect is also a necessary part of IE in many policy settings. If the purpose of an IE is accountability rather than learning, however, explanation may be less important.
- IE should as far as possible reflect the normative principles of development aid, and these have practical implications for IE. For example, working through partners leads to multi-stage, indirect causal chains that IE has to analyse, and using a country's own systems can limit access to certain kinds of data.
- Appropriate IE designs should match the evaluation question being asked and the attributes of programmes. Attributes might include duration and time scale, nonlinearity and unpredictability, local customisation of programmes, indirect delivery, and multiple interventions that influence each other.
- There are a range of potential designs based on a number of causal inference logics that are suitable for use in IE. Such designs include experimental, statistical, theory based, case-based and participatory approaches. Studies of 'cases' that combine within-case analysis and comparisons across cases are especially suited to IE in complex settings.
- Designs and methods applied to narrowly specified interventions can support strong causal claims, but as the scope and scale of an intervention increases the strength of causal claims is reduced. Many programmes are broad in scope and made up of many sub-programmes. Policymakers may therefore have to accept a trade-off between strong causal inference and relevance. There are some remedies to this dilemma: combining methods to strengthen causal inference, and applying 'tests' to evidence generated in broadly specified programmes.
- Most interventions are a 'contributory cause' and part of a causal package. Other causal elements might include other country-based policies and programmes, institutional and cultural pre-conditions, and exogenous factors like an economic cycle or world trade. (Theory-based and case-based approaches are especially suited to unpicking how causal factors combine and what might be the contribution of an intervention. However such approaches are not good at estimating the quantity or extent of a contribution.) Regarding evaluation questions, asking: 'Did the intervention make a difference?' allows space for combinations of causes, whereas asking 'Did the intervention work?' expects the intervention to be a sole cause.
- Combining methods is a useful strategy both to increase confidence in IE findings and help compensate for the weaknesses of particular methods.
- A Quality Assurance framework is needed to assure policy makers that IE findings are defensible and to encourage evaluators to improve IE design and implementation. It is possible to use a common QA system across different designs and methods. A three-part framework for QA is proposed that covers: the conduct of an evaluation over the life-cycle; technical criteria to judge IE designs and methods; and normative criteria that follow from the way the aid relationship is understood post Paris, Accra and Busan. This framework uses standards such as Reliability, Robustness, Transparency, Validity and Rigour in ways that can be applied to a broad range of designs.
Access full text: available online
Stern, E. et al. (2012) 'Broadening the Range of Designs and Methods for Impact Evaluations', DFID Working Paper 38, London: Department for International Development
Organisation: Department for International Development (DFID), http://www.dfid.gov.uk