Impact evaluation
Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones. In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question: how would outcomes such as participants' well-being have changed if the intervention had not been undertaken? This involves counterfactual analysis, that is, "a comparison between what actually happened and what would have happened in the absence of the intervention." Impact evaluations seek to answer cause-and-effect questions. In other words, they look for the changes in outcome that are directly attributable to a program.
Impact evaluation helps people answer key questions for evidence-based policy making: what works, what doesn't, where, why and for how much? It has received increasing attention in policy making in recent years in the context of both Western and developing countries. It is an important component of the armory of evaluation tools and approaches and integral to global efforts to improve the effectiveness of aid delivery and public spending more generally in improving living standards. Originally more oriented towards evaluation of social sector programs in developing countries, notably conditional cash transfers, impact evaluation is now being increasingly applied in other areas such as the agriculture, energy and transport.
Counterfactual evaluation designs
analysis enables evaluators to attribute cause and effect between interventions and outcomes. The 'counterfactual' measures what would have happened to beneficiaries in the absence of the intervention, and impact is estimated by comparing counterfactual outcomes to those observed under the intervention. The key challenge in impact evaluation is that the counterfactual cannot be directly observed and must be approximated with reference to a comparison group. There are a range of accepted approaches to determining an appropriate comparison group for counterfactual analysis, using either prospective or retrospective evaluation design. Prospective evaluations begin during the design phase of the intervention, involving collection of baseline and end-line data from intervention beneficiaries and non-beneficiaries ; they may involve selection of individuals or communities into treatment and comparison groups. Retrospective evaluations are usually conducted after the implementation phase and may exploit existing survey data, although the best evaluations will collect data as close to baseline as possible, to ensure comparability of intervention and comparison groups.There are five key principles relating to internal validity and external validity which rigorous impact evaluations should address: confounding factors, selection bias, spillover effects, contamination, and impact heterogeneity.
- Confounding occurs where certain factors, typically relating to socioeconomic status, are correlated with exposure to the intervention and, independent of exposure, are causally related to the outcome of interest. Confounding factors are therefore alternate explanations for an observed relationship between intervention and outcome.
- Selection bias, a special case of confounding, occurs where intervention participants are non-randomly drawn from the beneficiary population, and the criteria determining selection are correlated with outcomes. Unobserved factors, which are associated with access to or participation in the intervention, and are causally related to the outcome of interest, may lead to a spurious relationship between intervention and outcome if unaccounted for. Self-selection occurs where, for example, more able or organized individuals or communities, who are more likely to have better outcomes of interest, are also more likely to participate in the intervention. Endogenous program selection occurs where individuals or communities are chosen to participate because they are seen to be more likely to benefit from the intervention. Ignoring confounding factors can lead to a problem of omitted variable bias. In the special case of selection bias, the endogeneity of the selection variables can cause simultaneity bias.
- Spillover occurs when members of the comparison group are affected by the intervention.
- Contamination occurs when members of treatment and/or comparison groups have access to another intervention which also affects the outcome of interest.
- Impact heterogeneity refers to differences in impact due by beneficiary type and context. High quality impact evaluations will assess the extent to which different groups benefit from an intervention as well as the potential effect of context on impact. The degree that results are generalizable will determine the applicability of lessons learned for interventions in other contexts.
Experimental design
Under experimental evaluations the treatment and comparison groups are selected randomly and isolated both from the intervention, as well as any interventions which may affect the outcome of interest. These evaluation designs are referred to as randomized control trials. In experimental evaluations the comparison group is called a control group. When randomization is implemented over a sufficiently large sample with no contagion by the intervention, the only difference between treatment and control groups on average is that the latter does not receive the intervention. Random sample surveys, in which the sample for the evaluation is chosen randomly, should not be confused with experimental evaluation designs, which require the random assignment of the treatment.The experimental approach is often held up as the 'gold standard' of evaluation. It is the only evaluation design which can conclusively account for selection bias in demonstrating a causal relationship between intervention and outcomes. Randomization and isolation from interventions might not be practicable in the realm of social policy and may be ethically difficult to defend, although there may be opportunities to use natural experiments. Bamberger and White highlight some of the limitations to applying RCTs to development interventions. Methodological critiques have been made by Scriven on account of the biases introduced since social interventions cannot be fully blinded, and Deaton has pointed out that in practice analysis of RCTs falls back on the regression-based approaches they seek to avoid and so are subject to the same potential biases. Other problems include the often heterogeneous and changing contexts of interventions, logistical and practical challenges, difficulties with monitoring service delivery, access to the intervention by the comparison group and changes in selection criteria and/or intervention over time. Thus, it is estimated that RCTs are only applicable to 5 percent of development finance.
Quasi-experimental design
al approaches can remove bias arising from selection on observables and, where panel data are available, time invariant unobservables. Quasi-experimental methods include matching, differencing, instrumental variables and the pipeline approach; they are usually carried out by multivariate regression analysis.If selection characteristics are known and observed, they can be controlled for to remove the bias. Matching involves comparing program participants with non-participants based on observed selection characteristics. Propensity score matching uses a statistical model to calculate the probability of participating on the basis of a set of observable characteristics and matches participants and non-participants with similar probability scores. Regression discontinuity design exploits a decision rule as to who does and does not get the intervention to compare outcomes for those just either side of this cut-off.
Difference in differences or double differences, which use data collected at baseline and end-line for intervention and comparison groups, can be used to account for selection bias under the assumption that unobservable factors determining selection are fixed over time.
Instrumental variables estimation accounts for selection bias by modelling participation using factors that are correlated with selection but not the outcome, thus isolating the aspects of program participation which can be treated as exogenous.
The pipeline approach uses beneficiaries already chosen to participate in a project at a later stage as the comparison group. The assumption is that as they have been selected to receive the intervention in the future they are similar to the treatment group, and therefore comparable in terms of outcome variables of interest. However, in practice, it cannot be guaranteed that treatment and comparison groups are comparable and some method of matching will need to be applied to verify comparability.
Non-experimental design
Non-experimental impact evaluations are so-called because they do not involve a comparison group that does not have access to the intervention. The method used in non-experimental evaluation is to compare intervention groups before and after implementation of the intervention. Intervention interrupted time-series evaluations require multiple data points on treated individuals before and after the intervention, while before versus after designs simply require a single data point before and after. Post-test analyses include data after the intervention from the intervention group only. Non-experimental designs are the weakest evaluation design, because to show a causal relationship between intervention and outcomes convincingly, the evaluation must demonstrate that any likely alternate explanations for the outcomes are irrelevant. However, there remain applications to which this design is relevant, for example, in calculating time-savings from an intervention which improves access to amenities. In addition, there may be cases where non-experimental designs are the only feasible impact evaluation design, such as universally implemented programmes or national policy reforms in which no isolated comparison groups are likely to exist.Biases in estimating programme effects
Randomized field experiments are the strongest research designs for assessing program impact. This particular research design is said to generally be the design of choice when it is feasible as it allows for a fair and accurate estimate of the program's actual effects.With that said, randomized field experiments are not always feasible to carry out and in these situations there are alternative research designs that are at the disposal of an evaluator. The main problem though is that regardless of which design an evaluator chooses, they are prone to a common problem: Regardless of how well thought through or well implemented the design is, each design is subject to yielding biased estimates of the program effects. These biases play the role of exaggerating or diminishing program effects. Not only that, but the direction the bias may take cannot usually be known in advance. These biases affect the interest of the stakeholder. Furthermore, it is possible that program participants are disadvantaged if the bias is in such a way that it contributes to making an ineffective or harmful program seem effective. There is also the possibility that a bias can make an effective program seem ineffective or even as far as harmful. This could possibly make the accomplishments of program seem small or even insignificant therefore forcing the personnel and even cause the program's sponsors to reduce or eliminate the funding for the program.
It is safe to say that if an inadequate design yields bias, the stakeholders who are largely responsible for the funding of the program will be the ones most concerned; the results of the evaluation help the stakeholders decide whether or not to continue funding the program because the final decision lies with the funders and the sponsors. Not only are the stakeholders mostly concerned, but those taking part in the program or those the program is intended to positively affect will be affected by the design chosen and the outcome rendered by that chosen design. Therefore, the evaluator's concern is to minimize the amount of bias in the estimation of program effects.
Biases are normally visible in two situations: when the measurement of the outcome with program exposure or the estimate of what the outcome would have been without the program exposure is higher or lower than the corresponding "true" value. Unfortunately, not all forms of bias that may compromise impact assessment are obvious.
The most common form of impact evaluation design is comparing two groups of individuals or other units, an intervention group that receives the program and a control group that does not. The estimate of program effect is then based on the difference between the groups on a suitable outcome measure. The random assignment of individuals to program and control groups allows for making the assumption of continuing equivalence. Group comparisons that have not been formed through randomization are known as non-equivalent comparison designs.
Selection bias
When there is an absence of the assumption of equivalence, the difference in outcome between the groups that would have occurred regardless creates a form of bias in the estimate of program effects. This is known as selection bias. It creates a threat to the validity of the program effect estimate in any impact assessment using a non-equivalent group comparison design and appears in situations where some process responsible for influences that are not fully known selects which individuals will be in which group instead of the assignment to groups being determined by pure chance. This may be because of participant self-selection, or it may be because of program placement.Selection bias can occur through natural or deliberate processes that cause a loss of outcome data for members of the intervention and control groups that have already been formed. This is known as attrition and it can come about in two ways : targets drop out of the intervention or control group cannot be reached or targets refuse to co-operate in outcome measurement. Differential attrition is assumed when attrition occurs as a result of something either than explicit chance process. This means that "those individuals that were from the intervention group whose outcome data are missing cannot be assumed to have the same outcome-relevant characteristics as those from the control group whose outcome data are missing". However, random assignment designs are not safe from selection bias which is induced by attrition.
Other forms of bias
There are other factors that can be responsible for bias in the results of an impact assessment. These generally have to do with events or experiences other than receiving the program that occur during the intervention. These biases include secular trends, interfering events and maturation.Secular trends or secular drift
Secular trends can be defined as being relatively long-term trends in the community, region or country. These are also termed secular drift and may produce changes that enhance or mask the apparent effects of a. For example, when a community's birth rate is declining, a program to reduce fertility may appear effective because of bias stemming from that downward trend.Interfering events
Interfering events are similar to secular trends; in this case it is the short-term events that can produce changes that may introduce bias into estimates of program effect, such as a power outage disrupting communications or hampering the delivery of food supplements may interfere with a nutrition program.Maturation
Impact evaluation needs to accommodate the fact that natural maturational and developmental processes can produce considerable change independently of the program. Including these changes in the estimates of program effects would result in bias estimates. An example of this form of bias would be a program to improve preventative health practices among adults may seem ineffective because health generally declines with age."Careful maintenance of comparable circumstances for program and control groups between random assignment and outcome measurement should prevent bias from the influence of other differential experiences or events on the groups. If either of these conditions is absent from the design, there is potential for bias in the estimates of program effect".
Estimation methods
Estimation methods broadly follow evaluation designs. Different designs require different estimation methods to measure changes in well-being from the counterfactual. In experimental and quasi-experimental evaluation, the estimated impact of the intervention is calculated as the difference in mean outcomes between the treatment group and the control or comparison group. This method is also called randomized control trials. According to an interview with Jim Rough, former representative of the American Evaluation Association, in the magazine D+C Development and Cooperation, this method does not work for complex, multilayer matters. The single difference estimator compares mean outcomes at end-line and is valid where treatment and control groups have the same outcome values at baseline. The difference-in-difference estimator calculates the difference in the change in the outcome over time for treatment and comparison groups, thus utilizing data collected at baseline for both groups and a second round of data collected at end-line, after implementation of the intervention, which may be years later.Impact Evaluations which have to compare average outcomes in the treatment group, irrespective of beneficiary participation, to outcomes in the comparison group are referred to as intention-to-treat analyses. Impact Evaluations which compare outcomes among beneficiaries who comply or adhere to the intervention in the treatment group to outcomes in the control group are referred to as treatment-on-the-treated analyses. ITT therefore provides a lower-bound estimate of impact, but is arguably of greater policy relevance than TOT in the analysis of voluntary programs.
Debates
While there is agreement on the importance of impact evaluation, and a consensus is emerging around the use of counterfactual evaluation methods, there has also been widespread debate in recent years on both the definition of impact evaluation and the use of appropriate methods.Definitions
The International Initiative for Impact Evaluation defines rigorous impact evaluations as: "analyses that measure the net change in outcomes for a particular group of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question that is being investigated and to the specific context".According to the World Bank's DIME Initiative, "Impact evaluations compare the outcomes of a program against a counterfactual that shows what would have happened to beneficiaries without the program. Unlike other forms of evaluation, they permit the attribution of observed changes in outcomes to the program being evaluated by following experimental and quasi-experimental designs".
Similarly, according to the US Environmental Protection Agency impact evaluation is a form of evaluation that assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of a program.
According to the World Bank's Independent Evaluation Group, impact evaluation is the systematic identification of the effects positive or negative, intended or not on individual households, institutions, and the environment caused by a given development activity such as a program or project.
Impact evaluation has been defined differently over the past few decades. Other interpretations of impact evaluation include:
- An evaluation which looks at the impact of an intervention on final welfare outcomes, rather than only at project outputs, or a process evaluation which focuses on implementation;
- An evaluation carried out some time after the intervention has been completed so as to allow time for impact to appear; and
- An evaluation considering all interventions within a given sector or geographical area.
Common definitions of 'impact' used in evaluation generally refer to the totality of longer-term consequences associated with an intervention on quality-of-life outcomes. For example, the Organization for Economic Cooperation and Development's Development Assistance Committee defines impact as the "positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended". A number of international agencies have also adopted this definition of impact. For example, UNICEF defines impact as "The longer term results of a program – technical, economic, socio-cultural, institutional, environmental or other – whether intended or unintended. The intended impact should correspond to the program goal." Similarly, Evaluationwiki.org defines impact evaluation as an evaluation that looks beyond the immediate results of policies, instruction, or services to identify longer-term as well as unintended program effects.
Technically, an evaluation could be conducted to assess 'impact' as defined here without reference to a counterfactual. However, much of the existing literature (e.g. NONIE Guidelines on Impact Evaluation adopts the OECD-DAC definition of impact while referring to the techniques used to attribute impact to an intervention as necessarily based on counterfactual analysis.
What is missing from the term 'impact' evaluation is the way 'impact' shows up long-term. For instance, most Monitoring and Evaluation 'logical framework' plans have inputs-outputs-outcomes and... impacts. While the first three appear during the project duration itself, impact takes far longer to take place. For instance, in a 5-year agricultural project, seeds are inputs, farmers trained in using them our outputs, changes in crop yields as a result of the seeds being planted properly in an outcome and families being more sustainably food secure over time is an impact. Such are very rare. They are also called ex-post evaluations or we are coining the term . While hundreds of thousands of documents call for them, rarely do donors have the funding flexibility - or interest - to return to see how sustained, and durable our interventions remained after project close out, after resources were withdrawn. There are many and how to foster .
Methodological debates
There is intensive debate in academic circles around the appropriate methodologies for impact evaluation, between proponents of experimental methods on the one hand and proponents of more general methodologies on the other. William Easterly has referred to this as . Proponents of experimental designs, sometimes referred to as 'randomistas', argue randomization is the only means to ensure unobservable selection bias is accounted for, and that building up the flimsy experimental evidence base should be developed as a matter of priority. In contrast, others argue that randomized assignment is seldom appropriate to development interventions and even when it is, experiments provide us with information on the results of a specific intervention applied to a specific context, and little of external relevance. There has been criticism from evaluation bodies and others that some donors and academics over-emphasize favoured methods for impact evaluation, and that this may in fact hinder learning and accountability. In addition, there has been a debate around the appropriate role for qualitative methods within impact evaluations.Theory-based impact evaluation
While knowledge of effectiveness is vital, it is also important to understand the reasons for effectiveness and the circumstances under which results are likely to be replicated. In contrast with 'black box' impact evaluation approaches, which only report mean differences in outcomes between treatment and comparison groups, theory-based impact evaluation involves mapping out the causal chain from inputs to outcomes and impact and testing the underlying assumptions. Most interventions within the realm of public policy are of a voluntary, rather than coercive nature. In addition, interventions are often active rather than passive, requiring a greater rather than lesser degree of participation among beneficiaries and therefore behavior change as a pre-requisite for effectiveness. Public policy will therefore be successful to the extent that people are incentivized to change their behaviour favourably. A theory-based approach enables policy-makers to understand the reasons for differing levels of program participation and the processes determining behavior change. Theory-Based approaches use both quantitative and qualitative data collection, and the latter can be particularly useful in understanding the reasons for compliance and therefore whether and how the intervention may be replicated in other settings. Methods of qualitative data collection include focus groups, in-depth interviews, participatory rural appraisal and field visits, as well as reading of anthropological and political literature.White advocates more widespread application of a theory-based approach to impact evaluation as a means to improve policy relevance of impact evaluations, outlining six key principles of the theory-based approach:
- Map out the causal chain which explains how the intervention is expected to lead to the intended outcomes, and collect data to test the underlying assumptions of the causal links.
- Understand context, including the social, political and economic setting of the intervention.
- Anticipate heterogeneity to help in identifying sub-groups and adjusting the sample size to account for the levels of disaggregation to be used in the analysis.
- Rigorous evaluation of impact using a credible counterfactual.
- Rigorous factual analysis of links in the causal chain.
- Use mixed methods.
Examples
More recently, impact evaluation has been applied to a range of interventions across social and productive sectors. 3ie has launched an online covering studies conducted in low- and middle income countries. Other organisations publishing Impact Evaluations include , the World Bank's and . The IEG of the World Bank has systematically assessed and summarized the experience of ten impact evaluation of development programs in various sectors carried out over the past 20 years.
Organizations promoting impact evaluation of development interventions
In 2006, the Evaluation Gap Working Group argued for a major gap in the evidence on development interventions, and in particular for an independent body to be set up to plug the gap by funding and advocating for rigorous impact evaluation in low- and middle-income countries. The was set up in response to this report. 3ie seeks to improve the lives of poor people in low- and middle-income countries by providing, and summarizing, evidence of what works, when, why and for how much. 3ie operates a grant program, financing impact studies in low- and middle-income countries and synthetic reviews of existing evidence updated as new evidence appears, and supports quality impact evaluation through its quality assurance services.Another initiative devoted to the evaluation of impacts is the . COSA is a non-profit global consortium of institutions, sustained in partnership with the International Institute for Sustainable Development Sustainable Commodity Initiative, the United Nations Conference on Trade and Development, and the United Nations International Trade Centre. COSA is developing and applying an independent measurement tool to analyze the distinct social, environmental and economic impacts of agricultural practices, and in particular those associated with the implementation of specific sustainability programs. The focus of the initiative is to establish global indicators and measurement tools which farmers, policy-makers, and industry can use to understand and improve their sustainability with different crops or agricultural sectors. COSA aims to facilitate this by enabling them to accurately calculate the relative costs and benefits of becoming involved in any given sustainability initiative.
A number of additional organizations have been established to promote impact evaluation globally, including , the , the World Bank's Development Impact Evaluation Initiative, the of the CGIAR, and the .