Along this continuum, the preference in usaids evaluation policy is clear. Having a manual that covers sample size calculations for quasiexperimental designs would also be useful. Power calculation for causal inference in social science. Evaluation theory, design and methods tei the evaluators. An impact evaluation design prepared once the intervention has started, and possibly been completed. Having a manual that covers sample size calculations for quasi experimental designs would also be useful. Impact evaluation designs are identified by the type of methods used to generate the counterfactual and can be broadly classified into three categories experimental, quasiexperimental and nonexperimental designs that vary in feasibility, cost, involvement during design or after implementation phase of the intervention, and degree of. An evaluation may examine the impact of an intervention on crime levels or whether an intervention has changed the perception of individuals. Quasi experimental designs identify a comparison group that is as similar as possible to the intervention group in terms of baseline preintervention characteristics. That is, obtain evidence that the program caused the outcomes observed in beneficiaries. Quasi experimental methods can be used retrospectively, i. Quasiexperimental studies are research designs that evaluate human behavior. This document is part of the better evidence in action toolkit. Quasiexperimental research designs are the most widely used research approach employed to evaluate the outcomes of social work programs and policies.
It may also be necessary to use quasiexperimental designs for exante. A quasiexperimental controlled evaluation of the impact of a hospice rapid response community service for endoflife care on achievement of preferred place of death laura m holdsworth, heather gage, simon coulton, annette king, and claire butler. Regression discontinuity design technical track impact evaluation workshop gdansk, february 2017. Introduction to impact evaluation design usaid learning lab. Quasiexperimental designs chris fifeschaw contents 4. Study designs for program evaluation university of pittsburgh.
A quasiexperimental impact evaluation of a criminal. These flexible methods can be adapted to a wide range of public health evaluation studies, and can incorporate assumptions and previous knowledge on the association of interest. Quasiexperimental research shares similarities with the traditional experimental design or randomized controlled trial, but it specifically lacks the element of random assignment to treatment or control. A matchedcomparison group design allows the evaluator to make causal claims about the impact of aspects of an intervention without having to randomly assign participants. Mar 09, 2016 the webinars were based on the impact evaluation series a userfriendly package of methodological briefs and four animated videos and presented by the briefs authors. Sociologists and psychologists often perform quasi experimental research to study individuals within the same environment.
Quasiexperimental design and methods unicef innocenti. It may also be necessary to use quasi experimental designs for exante. Quasiexperimental design research methods knowledge base. In conclusion, quasiexperimental designs represent a useful tool for public health evaluation, as well illustrated in the article by harper and colleagues. Pdf impact evaluation of seasonal malaria chemoprevention. Experimental and quasiexperimental designs involve comparisons. Developed using the evaluation plan template, the plan is for a quasiexperimental design qed. Experimental and quasi experimental study designs are often utilized in summative evaluations. Quasiexperimental design and methods better evaluation. The place impact evaluation comprised two quasi experimental designs to determine whether the implementation of oakland ceasefire was associated with citywide reductions in gun homicide. Although we focus on vaccination, the methods described can equally be applied to evaluations of other infectious disease interventions that act at a population level. Experimental, quasiexperimental or a descriptive and study. For impact evaluations, experimental methods generate the strongest evidence.
Quasi experimental research designs are the most widely used research approach employed to evaluate the outcomes of social work programs and policies. For an evaluation to have internal validity, the outcome of the control group needs to be indeed a. A matchedcomparison group design allows the evaluator to make causal claims about the impact of aspects of an intervention without having to. A quasiexperimental study to assess the impact of four. Often it is hard to recognize or justify spending on evaluation when it does not directly impact the programs community or has a large lagtime. In a quasiexperimental approach nonrandom methods of assignment are used. Methods of data analysis used in quasiexperimental designs may be expost single difference or. However, care must be taken about generalizing from a specific context. Sociologists and psychologists often perform quasiexperimental research to. Evaluation designs slides corporation for national and. Randomisation and quasiexperimental designs that establish control. Timing of intervention and data collection for impact evaluations with a. Quasiexperimental study comparing the proportion of swaps made by an intervention group 267 families who had signed up to the smart swaps campaign promoted through various media, including television and radio advertising in early 2014 and a comparison group 5 families resident in wales, signed up for change4life materials, but not directly exposed to the smart.
This study evaluates our mentored training program in terms of scientific productivity among applicants. This means there may be systematic differences between the groups. Quasiexperimental designs typically allow the researcher to control the assignment to the treatment condition but using some criterion other than random assignment e. A quasiexperimental study to assess the impact of four adolescent sexual health interventions in subsaharan africa context. A quasiexperimental controlled evaluation of the impact. These methods can be used to attribute change at any program or project.
Example evaluation plan for a quasiexperimental design. However, in a quasi experimental approach the target population is not randomly allocated to the intervention and comparison groups. The evaluation is funded through research grants from the soros foundation and the international growth centre. Impact evaluation designs program cycle project starter.
In this brief, we describe varied types of quasiexperimental evaluations and the. Quasiexperimental methods require more assumptions randomized assignment differencein. Purpose of impact evaluation impact evaluation serves both objectives of evaluation. A quasi experimental design is an empirical study, almost like an experimental design but without random assignment. This document provides an example of a detailed evaluation plan for evaluating the effectiveness of an intervention. Quasiexperimental methods that involve the creation of a comparison group are most often used when it is not possible to randomize individuals or groups to treatment and control groups. A quasiexperiment is an empirical interventional study used to estimate the causal impact of an intervention on target population without random assignment. In conclusion, quasi experimental designs represent a useful tool for public health evaluation, as well illustrated in the article by harper and colleagues. The welfare act of 1996 banned welfare and food stamp eligibility for felony drug offenders and gave states the ability to modify their use of the law. A quasiexperimental impact evaluation joseph kofi teye, louis boakyeyiadom, edward asiedu, julie litchfield, johnson wilson appiah kubi and mariama awumbila working paper 62 november 2019. What is meant by evaluating the impact of a program. Experimental and quasiexperimental evaluations 41 hannum.
My mentor, don campbell, often referred to them as queasy experiments because they give the experimental purists a. Developed using the evaluation plan template, the plan is for a quasi experimental design qed. This is always the case for expost impact evaluation designs. Quasi experimental methods that involve the creation of a comparison group are most often used when it is not possible to randomize individuals or groups to treatment and control groups. It may also be necessary to use quasi experimental designs for exante impact evaluations, for example, where ethical, political or logistical constraints, like the need for a phased geographical rollout, rule out randomization. In this paper, we discuss what a quasi experimental design is, when they might be used for vaccine evaluation, their strengths and limitations, and examples of their application. Before applying either experimental or quasiexperimental methods, evaluators must lay out the. Unless there was random assignment then a quasi experimental design has to be used.
Your intended evaluation design should be based on and aligned with the following. Impact evaluations are characterized by a specific evaluation design quasiexperimental or experimental in order to answer a causeandeffect question. Attention will also be given to issues related to the measurement of outcomes, ensuring detection of meaningful program effects, and interpreting. Effects of a food hub initiative in a disadvantaged. Evaluate your program and demonstrate program effectiveness. The basic structure of a quasiexperimental evaluation involves examining the impact of an intervention by taking measurements before and after it is implemented. Rigorous evaluations are needed to assess whether adolescent sexual health interventions have an effect on young peoples riskrelated perceptions and behaviors. The impact of migration on the welfare of households left behind in rural ghana. Mentored training and its association with dissemination.
A quasi experiment is an empirical interventional study used to estimate the causal impact of an intervention on target population without random assignment. Therefore evaluative research remains an underutilized tool. This document provides an example of a detailed evaluation plan for evaluating the. Quasiexperimental approaches are similar in that they compare units that are part of a development intervention with those that are not. The impact study will used a quasiexperimental design to assess the impact of. The place impact evaluation comprised two quasiexperimental designs to determine whether the implementation of oakland ceasefire was associated with citywide reductions in gun homicide.
Quasi experimental studies are research designs that evaluate human behavior. A quasiexperimental evaluation of the impact of public. This convergence of rising need for impact evaluation evidence among. Ie technical note united states agency for international. Both experimental and quasi experimental designs can produce credible impact evaluation findings, but there is a difference, and their classifications signal what that difference is.
My mentor, don campbell, often referred to them as queasy experiments because they give the experimental purists a queasy feeling. Seasonal malaria chemoprevention smc for children pdf 834. The criminal justice pilot paralegals solicit criminal cases through several strategies, including regular. Today, many states are revisiting their use of this ban, searching for ways to decrease the size of their prison populations. Impact evaluations need to account for the counterfactual what would have occurred without the intervention through the use of an experimental or quasiexperimental design using comparison and treatment groups. White, h and sabarwal, s 2014 quasiexperimental design and methods, methodological briefs.
Quasiexperimental evaluations part 6 in a series on practical evaluation methods kristin anderson moore, ph. Program participants themselvesbefore participating in the program. Quasiexperimental impact evaluation expost of ifads ad2m project in madagascar final report august 2017 hannah ring, american institutes for research mitchell morey, american institutes for research erin kavanagh, american institutes for research kevin kamto, american institutes for research nancy mccarthy, lead analytics. Overview although experimental studies are described as the gold standard for assessing the effectiveness of a program in changing outcomes, in some cases, quasiexperimental studies may be more feasible or appropriate. Quasiexperimental evaluations of social and public health interventions often face a risk of selection biasthe chance that observed program impacts reflect. A quasiexperimental design is one that looks a bit like an experimental design but lacks the key ingredient random assignment. Impact evaluation is an assessment of how the intervention being evaluated affects outcomes, whether these effects are intended or unintended.
Before applying either experimental or quasi experimental methods, evaluators must lay out the. The impact of migration on the welfare of households left. A quasiexperimental impact evaluation download wp62teyeetal2019theimpactofmigrationonthewelfareofhouseholdsleftbehindinruralghana. Excerpt quasiexperimental research designs, like experimental designs, test causal hypotheses. Use of quasiexperimental designs for vaccine evaluation. Quasiexperimental impact evaluation of ifads ad2m project. Rigorous impact evaluations may be undertaken for development objectives.
Evaluation design is the structure that provides the information needed to answer each of your evaluation questions. External validity the extent to which the results of the impact evaluation apply to another time or place. Appropriate evaluation methods therefore depend on several overall criterion. Quasi experimental designs typically allow the researcher to control the assignment to the treatment condition but using some criterion other than random assignment e. Applicability of impact evaluation european commission. Experimental impact evaluation 4 evaluative research. A quasi experimental impact evaluation joseph kofi teye, louis boakyeyiadom, edward asiedu, julie litchfield, johnson wilson appiah kubi and mariama awumbila working paper 62 november 2019. A quasiexperimental design is an empirical study, almost like an experimental design but without random assignment. Quasi experimental research shares similarities with the traditional experimental design or randomized controlled trial, but it specifically lacks the element of random assignment to treatment or control.
We carried out a quasi experimental impact evaluation of social impacts of wmas, collecting data from 24 villages participating in 6 different wmas across two geographical regions, and 18. The comparison group captures what would have been the outcomes if the programmepolicy had not been implemented i. These designs are often referred to as patchedup research designs poister, 1978, and usually, they do not test all the causal linkages in a logic model. Report of a study commissioned by the department for. Both experimental and quasiexperimental designs can produce credible impact evaluation findings, but there is a difference, and their classifications signal what that difference is. C, controlled prepost design, in which the impact equals the. Outline of principles of impact evalation discussion dra. Experimental studies try to determine causality or correlation in so far as this is possible. The program was the only factor influencing changes in the outcome over time. Excerpt quasi experimental research designs, like experimental designs, test causal hypotheses. An impact evaluation relies on rigorous methods to determine the changes in outcomes which can be attributed to a specific intervention based on causeandeffect analysis. A, crosssectional, controlled design, where the impact equals the difference d 1 between the study group black circle and control group gray circle. Four evaluations were based on randomized controlled trials, and six on nonrandomized, quasiexperimental designs.
65 1471 225 643 1614 749 26 933 212 1285 1070 672 685 640 422 1551 1336 611 1036 1120 1248 1242 851 1012 181 196 488 1044 381 964