Document Library

Key Text Measuring Change and Results in Voice and Accountability Work

Author: Jeremy Holland and Allyson Thirkell
Date: 2009
Size: 82 pages (3.5 MB)

Access full text: available online

Summary

How can the often intangible results of Voice and Accountability (V&A) interventions be measured? This paper adapts DFID’s 'Capability, Accountability and Responsiveness' (CAR) governance framework for use with V&A work. It maps existing indicators onto this adapted framework, developing a menu of V&A indicators and data collection instruments. Measures need to take account of the costs as well as benefits of poor people's voices being heard, and should reveal the obstacles to poor people's engagement. V&A indicator data can effectively combine observable and measurable changes in behaviour with perception scoring of the quality of those changes.

Citizen voice and accountability (V&A) work has emerged as a priority in international development, yet evidence of its impact on development outcomes is sparse. There are many challenges to measuring change through V&A interventions, particularly as progress often involves intangible changes in power relationships. An OECD DAC evaluation on Citizens' Voice and Accountability (2008) identified a lack of good indicators for measuring change in V&A. Indicators are concrete, specific descriptions of what is measured to gauge whether interventions have produced change.

The following framework builds on DFID's CAR framework in order to make explicit links to development impacts. It accommodates change at the level of individual behaviour and practice, and in policy and legislature.

Capability is used to refer to formal institutions (policies, laws, political freedoms and oversight mechanisms) and informal institutions (socio-cultural norms) that provide the 'enabling environment' for effective voice and accountability.
Accountability refers to the demand side of accountability relations. Vertical accountability involves direct engagement with governments and other duty-bearers using political voice through democratic political processes, and with service providers using consumer voice. Horizontal accountability involves state institutions engaging in mutual scrutiny.
Responsiveness refers to the supply side of accountability relations - the vertical responsiveness of duty-bearers to citizens and the horizontal responsiveness of duty-bearers to oversight by other parts of the state.

V&A interventions can be read through the Logframe Results Chain, which involves an output level that defines and measures changes in behaviour and power relations. However, logframe-based measurement done badly can encourage linear, reductionist and technocratic thinking regarding interventions that are non-linear, unpredictable and highly politicised. The five basic elements of the Logframe Results Chain are inputs, processes, outputs, outcomes and impact. In each of these elements, there is a desired result, and indicator(s) are chosen to show whether or not it is being achieved.

Joint planning (with donors and government) is important in designing more holistic programmes that work on both the demand and supply sides. To avoid establishing parallel monitoring mechanisms, existing data sets and existing country-level government or donor processes should be used. It is also important to select indicators and mechanisms appropriate to the local social and political context. The following management issues checklist offers a guide to planning the measurement of V&A work.

Step 1: What change are you trying to achieve? Identify a clear purpose and a transparent set of effect assumptions. Involve stakeholders and use a participatory approach to identify problems, solutions and significant changes that can be measured.
Step 2: Which indicators will you develop to measure this change? Indicators should be able to test effect assumptions about project outputs, outcomes and impacts, and should be informed by collated evidence on what has worked in the past and why. Create systems that build on indicators already in use by governments or donors.
Step 3: What types of data will these indicators require? V&A behavioural change can be measured by: (1) quantitative data generated by observation or recall (for example, using random sampling of project beneficiaries); or (2) quantification of qualitative changes using perception scores (for example via the purposive samples of key informants. Include both qualitative and quantitative indicators.
Step 4: When do you collect the data to measure change? DFID logframe guidance stresses the importance of establishing a baseline and measuring change according to a set of milestones leading to an identified target. More frequent monitoring is justified if the behavioural changes targeted can change over a short time period, so that rapid course correction in project activities can be made.

Access full text: available online

Source: Holland, J. and Thirkell, A., with E. Trepanier and L. Earle, 2009, 'Measuring Change and Results in Voice and Accountability Work', Working Paper 34, Department for International Development, London
Organisation: Department for International Development (DFID), http://www.dfid.gov.uk