Tip:
Highlight text to annotate it
X
Experimental designs are often called the “gold standard” of political science research. There are two key reasons for this.
The first is random assignment of subjects to experimental and control groups. The second is introduction of the stimulus to only the
experimental group. Together these two facets of experimental designs provide us with the possibility of making strong claims of internal
validity. It turns out, however, that experimental designs are often impractical or irrelevant. For example, researchers often do not have control
over introduction of the stimulus. Researchers interested in the effects of welfare reform typically cannot randomly expose some welfare
recipients but not others to reformed policies. Furthermore, random assignment of units of analysis to experimental and control groups is
often beyond the control of researchers. Researchers interested in the association between the race of a defendant and the
likelihood of a death penalty sentence cannot assign the color of the skin of defendants who are arrested for death penalty-eligible crimes.
In this presentation, our focus is on two particularly important types of non- experimental designs—cross-sectional
designs and time-series designs. For both types of research designs, we will first lay out their key components and then encounter
practical examples of how these research designs have been applied by researchers interested in politics and public policy.
Let’s begin with the cross- sectional design and its five key characteristics. First, in cross-sectional designs, the researcher
does not have control over the introduction of the independent variable, which naturally occurs out there in the real world. Second, the
researcher does not, cannot, assign units to experimental and control groups. Third, the measurement of the independent and
dependent variables takes place at the same time, usually after they have occurred naturally. Fourth, control for the effects of other possible
stimuli is exercised not through randomization, but through statistics. Just what we mean by control via statistics will become apparent
toward the end of the course. For now, just note this characteristic of cross-sectional designs and move on. Fifth, time sequence must be
established through theory, as opposed to the experimental sequencing of pre-test, introduction of the stimulus, and post-test.
Now let’s bring these characteristics to life through an example of the cross-sectional design. Data reveal that 83% of
those on death row in Philadelphia are African American. Is race a decisive factor in determining whether eligible defendants get the
death penalty? Here is how one research design attempted to answer this question. Researchers took a sample of death penalty-
eligible murders in Pennsylvania that occurred between 1983 and 1993. The unit of analysis was the ***. The dependent variable was
whether the defendant received a death penalty sentence. The independent variable was the race of the defendant. It goes without saying that
lots of factors potentially affect whether the death penalty is imposed—the severity of the ***, the defendant’s criminal background,
the quality of the defendant’s attorney. The researchers found that, even when accounting for these other factors, the likelihood of
receiving the death penalty was 38% higher for African Americans than for defendants of other races.
Now why would this research design be considered an example of the cross- sectional design? Let’s go back to the five key
characteristics of cross-sectional designs First, there was no control over the introduction of the stimulus. The race of the defendant, the
independent variable in the analysis, cannot be manipulated. Second, there can be no assignment of murders to experimental and
control groups, as these murders simply occurred in the real world. Third, the measurement of the independent and
dependent variables took place at the same time, after the fact by collecting information about murders that had already occurred at an
earlier time. Fourth, the researchers controlled for factors such as the severity of the ***, the defendant’s criminal background, and the
quality of the defendant’s attorney through statistics, not via randomization. Fifth, the time sequencing was established through theory, not
through the sequencing of the pre-test, stimulus, and post-test. The theory is obvious and easy here, as the defendant’s race naturally
exists before a death penalty-eligible crime is committed.
As you consider this cross- sectional design, you might think specifically about how you would evaluate its internal and
external validity. In what ways might this analysis fall short by these two standards for judging research designs?
In the meantime, let’s move on to the other major type of non-experimental design that we will consider in this presentation—the ti
me-series design. Time-series designs have five key characteristics. The researcher does not have control over the introduction of the
independent variable. The researcher does not assign units of analysis to experimental and control groups. Note that these two
characteristics also describe cross-sectional designs. The absence of random assignment and control over the introduction of the stimulus
are characteristics of all non-experimental designs. Third, in time-series designs, numerous measurements of the dependent
variable are taken before and after the introduction of the independent variable. This may sound identical to experimental designs,
where the dependent variable is measured through the pre- and post-tests and the stimulus is introduced in between these tests.
But, as we will see in a moment, there are important differences that make time-series sequencing different from sequencing in
experiments. Fourth, control for the effects of other possible stimuli is exercised not through randomization, but through statistics. Fifth, the
focal point of time-series designs is one unit of analysis compared to itself over time.
Here is example of the time- series design at work. What accounted for the significant drop in the number of Americans
receiving welfare during the 1990s? During this period, welfare rolls decreased dramatically, by 20% between 1993 and 1997, for a grand total
of nearly 3 million fewer welfare recipients. Was this decline a product of a growing economy? Were there public policies that might have
made a difference during this time period? For example, during this time period, the Clinton administration encouraged states to experiment
with innovative approaches to welfare policy. In response to this encouragement, states adopted policies that, among other things,
placed time limits on welfare recipients and instituted requirements that recipients actively look for work while receiving benefits.
Let’s think about why studying the effects of the economy and public policy on the number of Americans receiving welfare
might be an example of the time-series research design. First, it is not possible to control the introduction of the two key stimuli of
economic growth and state policy. These phenomena simply occurred in the real world, and the best we can do is to observe their
occurrence. Second, it is not possible to assign units of analysis to experimental and control groups. Third, the dependent variable, the
number of Americans receiving welfare, can be measured at multiple points in time, both before and after the period of economic growth and
new state policies. Fourth, control for the effects of other possible stimuli on the number of Americans receiving welfare cannot be
exercised through randomization. Rather, statistics will have to be utilized. Fifth, the focal point can be the United States, as the unit of
analysis, compared to itself over time. Just such a time-series analysis was conducted by the Clinton administration’s Council of Economic
Advisors, which found that over 40% of the decline in welfare participation was due to economic expansion, almost one-third of the
decline was due to state policy innovation, and the remainder of the drop was due to other factors such as increases in child care
spending. Once again, as you consider this
me-series design, you might think specifically about how you would evaluate its internal and external validity. In what ways might the analysis
of the Council of Economic Advisors fall short by these two standards for judging research designs?
Addressing such questions will be a key element in evaluating the research designs you encounter in this course, other courses, at work,
in the news. This skill will also be of great use as you work toward creating your own research designs.