Consultative Impact Monitoring of Policy - CoIMPact - Principles and General Procedures
CoIMPact starts with the premise that the results will be useful for carrying the voices of ordinary people into the policy-making arena. The CoIMPact approach recognizes that to do this, simply producing reports is not sufficient. For this reason a comprehensive results dissemination exercise has to be incorporated as one of the “phases” of CoIMPact to open the space for discussion about policy reforms and changes. This is enhanced by the regular involvement in the exercise of those who are in a position to make policy changes, from the key central and line ministries as well as those delivering services to the poor at the decentral or provincial level.
Further, the CoIMPact exercises are not one-off events, but are rather multi-stage and recurring exercises, integrated in the existing Poverty Monitoring Systems of the country.
In the development of a Poverty Reduction Strategy Programme (PRSP), participation and consultation are key concerns. To date, most countries adopting a PRSP have held widespread consultations during the formulation of the strategies. Often this process has been of an ad hoc nature involving civil society and private sector actors, while neglecting elected officials in parliament and elsewhere on the understanding that these tend to be weak institutions. The participation of these organisations in the policy formulation process has generated a demand for the continued involvement of the ordinary people in the monitoring of policy. The experience of CoIMPact exercises carried out to date suggests the method is able to enhance local level participation.
Figure 2: CoIMPact communication and information flows
Participatory monitoring and evaluation, which involves direct consultation with the beneficiaries, particularly the poor, plays an important role in expanding the information base for poverty diagnostics (identifying who the poor are), monitoring policy implementation (what impact this is having on people) and in the retrospective evaluation of the policy.
In addition to the participation of the direct beneficiaries, the collaborating partners in the exercise can include staff of responsible ministries, relevant local Government and decentral Offices, representatives of NGOs / CSOs and research institutions. This supports improved coordination between different agencies in carrying out anti-poverty activities. Additionally, rather than being a strain on scarce resources, it can act as a means of strengthening the capacity of those involved in performing their official duties, by helping them to understand better the needs and priorities of the people they are to serve.
... participatory poverty assessments - PPA
While CoIMPact and Participatory Poverty Assessments (PPA) essentially belong to the same family and have the same overall intention – to continue to use the Voices of the Poor to reinforce policy messages for politicians, technocrats and the media – they differ in a number of ways. The most striking difference is CoIMPact’s reliance on in depth policy analysis before collecting any data. This has a number of benefits. During data collection, it ensures that those going to the field understand fully what they need to investigate and the areas they need to probe. In the report writing stage it means that the various reports can answer specific questions connected to the impact selected policies are having on poor people.
Other differences are the attention given to the institutionalisation of the exercise. Originally, PPAs were designed and supervised by the staff of donor organisations, and managed and implemented by NGOs or research institutions (something that continues to be the case in a number of PRSPs). CoIMPact has rather taken up the issues of institutionalisation, seeking to be embedded in the national policy-making structure. This is done with the express intention of improving the relationship to the policy process.
The third major difference between a traditional PPA and CoIMPact is the system of feedback loops, incorporating all levels of society from the individual through to decentral administration and national level policy makers. These are ensured by involving decision-makers throughout the exercise, including: the choice of the policy to be investigated, the specific design of each round and consultations during the policy analysis, as well as the field research and report writing. The CoIMPact approach considers that this arrangement:
Unlike many other data collection tools, such as household surveys, CoIMPact exercises pay a great deal of attention to issues of Gender at various levels of the exercise. Consideration of this issue needs to be made at virtually every step:
.... other monitoring tools
CoIMPact exercises are complimentary to other monitoring efforts. Most obviously, the use of other sources of data helps in the triangulation of results and informing survey design. However, the relationship between the various exercises can take on a number of forms:
Institutional set-up for carrying out CoIMPact
Agreeing on the institutional arrangements is one of the key prerequisites for the exercise. CoIMPact requires a strong institutional base for it to have any meaningful impact on policy. Experience to date has entailed focusing on government, while at the same time generating additional links to other institutions, such as NGOs and development partners. Involving a broad spread of actors helps to ensure that the results of the exercise are used in the reform of policy, and that the most relevant policies are investigated.
Within government, there is a need to focus on a strong central organisation, such as the Ministry of Planning, or the Ministry of Economic Planning and Development. The officials and staff of these institutions are to be involved at all stages, from conceptualisation to the final release of the report. However, the exercises have to involve others as well – including the national statistical offices, decentral government, key line ministries and non-governmental organisations. The key is to ensure that the exercise finds its home in an organisation that is committed to introducing, supporting and improving pro-poor policies (or programmes) in a cross-sectoral way and is in a position to push this agenda forward.
Using people from government institutions in the planning, implementation and analysis helps to improve the access to the policy process and generates additional support for the results at hand.
Providing Timely Information
One of the principles behind the CoIMPact approach is to ensure that information on the implementation of policies and indications of outcomes and impact can be provided expeditiously to allow mitigating interventions to be undertaken when necessary. Experience shows that it is possible to carry out a full round of a CoIMPact exercise (from conceptualisation to production of the report) within six months. However, it must be remembered that every country is different, and the first round of an exercise in a new environment may take more time – this can be justified if all the components are put in place, and the key stakeholders are brought “on board”, which eventually will lead to the use of the results of the exercise. Other issues that will have an effect on the length of time are the number of people involved in the data collection and the number of sites visited.
One of the principles behind the CoIMPact approach is to ensure that information on the implementation of policies and indications of outcomes and impact can be provided in a timely and quick manner, to allow interventions to be undertaken. Timing and sequencing should fit with the policy formulation cycle in order to ensure that the findings have an influence.
When to use CoIMPact
CoIMPact is designed to provide feedback to policy and decision makers on people’s perceptions about the outcomes and impact of policy intervention. For it to be successfully implemented anyone of the following conditions needs to be fulfilled:
Steps to carry out the CoIMPact process
Five broad phases are necessary in a CoIMPact exercise. The precise number of steps implemented in each phase will vary according to the needs of the country and implementing agency. Some key steps must be undertaken. However, they do not form a blueprint for carrying out the exercise. Each country is different and the exercise needs to reflect these differences. The watchwords then are flexibility not just in the planning stage but also in the implementation stage to allow for adaptation to any particular situation, and capacity building so that those taking part in the exercise are able to carry out future rounds.
1. The Preparatory Phase
The purpose of having a preparatory phase is to establish a smooth conceptual basis for the implementation of a round of the exercise, including reaching consensus on the objective, purpose and results so that they are appropriate to the prevailing national situation. At this stage, it is also important to clarify the roles and responsibilities of the key stakeholders and deal with issues of establishing the institutional framework for the exercise. Further, there are some practical issues that need to be addressed, including deciding on the coverage of the exercise (number of sites), the areas to be investigated (following widespread consultations), and an assessment of the financial costs, including a more general assessment of the resources necessary and available (including personnel). The most important steps include:
In the first iteration of the exercise, the preparation will require time. Occasionally this may seem disproportionate to the time spent on the other phases, however, in the long run the contribution it makes towards ownership and ensuring that the results find their way into the policy process justify the effort. Subsequent rounds of data collection might not need to spend as much time on the preparation phase of the exercise.
Figure 3: The CoIMPact process
2. The Policy Analysis Phase
The policy analysis phase examines the extent to which the priorities of the poor have found their way into policy documents, such as the Poverty Reduction Strategy Paper (PRSP) and whether (sector) policies are genuinely poverty-focused. At the same time, it identifies possible indicators from the policy documents against which success or impact can be assessed in a qualitative way. It results in the development of hypotheses for testing during the field phase. The policy analysis also helps to ensure that the findings of the field phase can be linked to policy interventions in the production of the final report. The policy analysis ties in with the preparation phase when key policy and decision-makers identify the information gaps they would like filled and select the areas for investigation. It is important to keep refining the policy analysis after each step, including:
The analysis requires a number of iterations to allow continuous finetuning and to ensure the incorporation of the views and needs of the broadening circle of stakeholders. Capacity building to carry out further analysis exercises needs to be one of the key objectives of this stage.
The output from this phase of the exercise is the development of a number of hypotheses to be tested. Questions to achieve this are included in a checklist for field-use. The checklist ensures that the key policy areas are addressed by the teams in the field in sufficient detail to make their analysis meaningful and to facilitate the comparison of reports from different sites, enabling the easier production of a final report at national level. Checklists/questionnaires will require testing prior field implementation; this requires time.
3. The Data Collection Phase
Prior to data collection the selection of the multi-disciplinary team has to be undertaken. A balance has to be found between use of government employees, freelancers, and civil society. The former can provide a link to policy, but may have time constraints. The latter may often want to participate, but also have time constraints, and may not have a direct link to policy making. Freelancers will have more time, but may not be able to provide the link to policy.
The data collection phase should collect reliable and relevant information about poverty, the priorities of the poor and the perception of these people about the impact that policies, designed for their benefit, actually are having on them. As well as the data collection, there are three key elements at this stage - training, testing checklists and methodologies, and documentation, even though paying adequate attention to administrative and management issues at this phase will help to contribute to a successful exercise. The steps of the data collection phase include:
The CoIMPact exercise is founded on the principal of “community immersion” that is, the team spends their time with the participating community. This allows the development of a greater sense of empathy between the two, and allows the team, through observation, to triangulate some of the findings, by seeing elements of the daily lives of the community. To facilitate this, a stay of between five days and a week with each community is recommended. Staying for such a length of time also eases some of the strains placed on the participants’ time – appointments, interviews and discussions can be carried out at their convenience.
4. The Data Analysis Phase
The ultimate goal of the analysis phase is to produce a series of reports that reflect the opinions of the communities visited in a manner that will lead to their voices being heard in the policy-making process at various levels. The analysis and report writing consists of processing the results of the field work so that it is meaningful in the context of the research topics and presenting it in a way that is useful to policy makers. The data analysis phase compares the Policy Analysis with the data collected in the communities in order to examine the impact of the policies and interventions being investigated. The biggest difficulty at this stage relates to drawing macro conclusions from micro-analysis at site report level. It is important in the final report to ensure that minor pieces of information are not used to exaggerate findings beyond their real significance. The acceptability of such data will very much depend on the depth of triangulation of findings in the research process, and re-examining controversial information. The major steps are:
The result of this phase will be the production of a comprehensive national level report, which includes policy related recommendations. The main findings of this report then need to be produced in a number of formats to help facilitate the dissemination phase.
5. The Dissemination Phase
The Dissemination Phase is designed to ensure that the findings and results of the exercise are shared with those who may be in a position to take some action on them. This is essential bearing in mind that the overall objective of the exercise is to ensure the voices of the poor are heard in the policy-making process and to keep the policy makers informed about progress in implementing and improving their policies. The list of stakeholders to be included at this stage can be a long one – including the sector ministries, the Parliament, local administrations, NGOs, international development organisations, the media, and the public at large. In this regard, CoIMPact exercises have focused on the production of an array of reports and briefings (products) – these can include:
Providing such an array of information helps to keep various groups of people informed about the impact that policies have on them, and helps government understand unexpected side effects that have emerged. This contributes to the debate on the effectiveness of policy and supports national dialogue on poverty.