1.13 - Experimental Design
Overview
Experiments are the only kind of study that can show cause and effect, and that power comes from good design, not from a large number of subjects. This topic covers the parts of a well-designed experiment (comparison, random assignment, replication, and control), placebos and blinding, the three experimental designs on the AP exam (completely randomized, randomized block, and matched pairs), and the conclusions an experiment can support. Terms like experimental unit, factor, treatment, and response variable come from Topic 1.10.
Elements of a well-designed experiment
A well-designed experiment includes all four of these:
Element | What it means | Why it matters |
Comparison | At least two treatment groups, one of which could be a control group | Shows whether the treatment of interest does better than something else |
Random assignment | Treatments are assigned to experimental units at random | Makes the groups alike in other ways, so differences in the response can be blamed on the treatments |
Replication | More than one experimental unit gets each treatment | Lets you tell a real treatment effect from chance differences between units |
Direct control | Potential extraneous sources of variation are kept the same for every unit | Removes variation that could hide the treatment effect |
Comparison and control groups
An experiment needs something to compare against. A control group is a collection of experimental units created for comparison. The control group may get no treatment, the current standard treatment, or an inactive treatment called a placebo, like a sugar pill that looks exactly like the real one.
Why not just give everyone the new treatment and see if they improve? Because people often improve anyway: colds go away, and allergy seasons end.
The placebo effect
The placebo effect is the difference between the average response to a placebo and the average response to no treatment. People often respond to a fake treatment just because they believe they're being treated. A placebo group gets that same boost, so comparing the real treatment to a placebo separates the treatment's actual effect from the placebo effect.
Example 1: Measuring the placebo effect
In an experiment on headaches, volunteers were randomly assigned to a new pain reliever, a placebo pill, or no pill at all. The table shows each group's mean pain reduction one hour later, on a 0 to 10 scale.
Group | Mean pain reduction (points) |
No pill | 0.9 |
Placebo pill | 2.1 |
New pain reliever | 3.4 |
Placebo effect: mean response to the placebo minus mean response to no treatment.
Effect of the pain reliever beyond the placebo:
Answer: taking a pill with no medicine in it reduced pain by 1.2 points more, on average, than taking nothing. If the researchers had compared the pain reliever only to no pill, they would have credited it with points of pain reduction, and almost half of that was the placebo effect. The comparison with the placebo group shows the medicine itself added about 1.3 points.
Blinding
Expectations can affect results from both sides. Subjects who know they got the real drug may feel better, and researchers who know may rate them as better. Blinding prevents this.
- Single-blind (single-masked): the participants don't know which treatment they're receiving, but the members of the research team who interact with them do, or the other way around.
- Double-blind (double-masked): neither the participants nor the members of the research team who interact with them know which treatment each participant is receiving. Someone outside the team keeps track of the assignments until the data are collected.
Blinding isn't always possible. Subjects comparing a running program with a yoga program know which one they're doing, but the people measuring their blood pressure can still be kept from knowing (single-blind).
Extraneous variables and confounding
An extraneous variable, also called an extraneous source of variation, is a variable that is known (or believed) to affect the response but isn't an explanatory variable being studied. In an allergy medicine experiment, the severity of each person's allergies, their age, and how much pollen they're exposed to are all extraneous variables.
A confounding variable in an experiment is related to the explanatory variable in such a way that it's hard to tell which one, the explanatory variable or the confounding variable, is causing the change in the response. A well-designed experiment reduces the potential for confounding variables.
Here's a design where confounding is a problem: a teacher lets students choose whether to use a new study app, then compares test scores. Students who choose the app are likely more motivated, and motivation raises test scores on its own. Motivation is confounded with the app, so any difference in scores can't be blamed on the app. The fix is random assignment.
Random assignment
The purpose of random assignment is to create treatment groups that are as similar as possible with respect to extraneous variables. If random assignment works, the distribution of each extraneous variable (age, allergy severity, motivation, even variables nobody thought to measure) will be about the same in every treatment group. Then, if the groups' responses differ by more than chance would explain, the treatments are the only reasonable explanation.
How to describe random assignment
Like describing an SRS, a complete description says exactly how chance decides each unit's treatment. For 24 volunteers and 2 treatments:
- Number the volunteers from 1 to 24.
- Use a random number generator (on a TI-84,
randInt(1, 24)) to pick 12 different numbers from 1 to 24, ignoring repeats.
- The volunteers with those numbers get the new medicine. The other 12 get the placebo.
Another way: write "medicine" on 12 identical slips of paper and "placebo" on 12 more, mix them in a hat, and have each volunteer draw one slip without replacement. Flipping a coin for each volunteer is also random assignment, but it won't always give equal group sizes.
Replication and direct control
Replication within an experiment means more than one experimental unit is assigned to each treatment. Units always differ a bit; one tomato plant just grows bigger than another. With only one plant per fertilizer, you couldn't tell a real fertilizer effect from plant-to-plant differences. With many plants per fertilizer, chance differences tend to average out.
Direct control means keeping the settings of potential extraneous sources of variation the same from one experimental unit to the next. In a fertilizer experiment, every plant gets the same kind of pot, the same soil, the same amount of water, and the same light. A variable that's held constant can't explain differences in the response.
Experimental designs
The rest of this page follows one study. A company wants to know whether its new allergy medicine lowers allergy symptoms compared to a placebo. It has 24 volunteers with seasonal allergies, and the response is each volunteer's symptom score after two weeks (0 to 20, where higher is worse).
Completely randomized design
In a completely randomized design, treatments are assigned to the experimental units completely at random. Often each treatment gets the same number of units, but the group sizes don't have to be equal.
Here, the 24 volunteers are randomly split into two groups of 12. One group takes the new medicine for two weeks, and the other takes a placebo that looks identical. Neither the volunteers nor the staff who record symptoms know who got which pill, so the experiment is double-blind. At the end, compare the mean symptom scores of the two groups.
Randomized block design
A blocking variable is a source of extraneous variation in the response. In a randomized block design, the experimental units are first grouped by similar values of the blocking variable. These groups are called blocks, and the units within a block are homogeneous with respect to the blocking variable. After the blocks are formed, the treatments are randomly assigned to units within each block, so every treatment appears in every block.
Allergy severity strongly affects symptom scores, so it makes a good blocking variable. Split the volunteers into a block of 12 with mild allergies and a block of 12 with severe allergies. Within each block, randomly assign 6 volunteers to the medicine and 6 to the placebo. Then compare the treatments within each block.
Why block?
The purpose of blocking is to separate the variation in the response caused by the blocking variable from the rest of the extraneous variation. That allows a more precise comparison of the treatments. Within a block, you can compare treatments without worrying about variation caused by the blocking variable.
The dotplots show possible results from the randomized block design (made-up data). Red dots are volunteers who took the medicine, and black dots are volunteers who took the placebo.
- Blocks ignored (top): the medicine scores run from 4 to 15, and the placebo scores run from 7 to 18. The means differ by 3 points (9.5 for the medicine and 12.5 for the placebo), but the groups overlap a lot, because most of the spread comes from the difference between mild and severe allergies. Each group's standard deviation is about 4.3.
- Within each block (bottom): in the mild block, the medicine group averages 5.5 and the placebo group 8.5. In the severe block, the medicine group averages 13.5 and the placebo group 16.5. Within a block, the standard deviation is only about 1.0, so the medicine's 3-point advantage is easy to see.
The medicine's effect is 3 points either way. Blocking removed the variation caused by allergy severity, so the effect stands out. A good rule: block on variables you know affect the response, and let random assignment handle the rest.
Matched pairs design
A matched pairs design is a randomized block design with only two treatments. It comes in two versions:
- Pairs of similar units: match the experimental units in pairs based on one or more extraneous variables, such as two volunteers with the same allergy severity and age. In each pair, randomly assign one member to each treatment, for example with a coin flip.
- Each unit gets both treatments: every experimental unit receives both treatments, and the order of the treatments is randomized. Each unit serves as its own match.
For the allergy study, the second version could work like this. Randomly assign 12 volunteers to take the medicine in weeks 1 and 2 and the placebo in weeks 4 and 5, and the other 12 to take them in the opposite order. Week 3 is a break with no pills, so the first pill's effect wears off. Then look at the difference between each volunteer's two scores.
The random order matters. If everyone took the medicine first and the pollen count rose over the season, the placebo would look worse just because of the timing.
Choosing an experimental design
One experimental design may be more appropriate than another based on the goals of the study, the characteristics of the population, and the sample and variables involved.
Design | Good choice when |
Completely randomized | The units are fairly similar, or you don't know of an extraneous variable that strongly affects the response |
Randomized block | A known variable strongly affects the response (like allergy severity, age, or field location), and you can group units by it before assigning treatments |
Matched pairs | There are exactly two treatments, and you can form closely matched pairs or give each unit both treatments |
Giving each unit both treatments only works when the first treatment won't affect the response to the second. You can't teach one student a brand-new skill two ways, because after the first lesson it isn't new.
Example 2: Choosing and describing a design (AP-style)
A sunscreen company wants to compare two formulas, A and B, for preventing sunburn. It has 30 volunteers whose skin types vary widely. Each volunteer will spend one hour in the sun, and then a dermatologist will rate the redness of the treated skin.
(a) Describe a matched pairs design for this experiment.
Each volunteer serves as their own pair. For each volunteer, flip a coin: heads means formula A goes on the left arm and formula B on the right arm, and tails means the reverse. (Randomizing the arm keeps a side that gets more sun from always favoring one formula.) After an hour in the sun, the dermatologist rates the redness of each arm without knowing which formula went where. For each volunteer, find the difference in redness (A minus B), and analyze the 30 differences.
(b) Why is this design better than a completely randomized design that gives formula A to 15 volunteers and formula B to the other 15?
Skin type strongly affects how easily people burn, and these volunteers vary a lot. In a completely randomized design, differences in skin type between the two groups would add a lot of variation to the redness ratings and could hide a real difference between the formulas. Comparing two arms on the same person keeps skin type (and time in the sun) the same within each pair, so the comparison of the formulas is more precise.
Conclusions from an experiment
Cause and effect: using random assignment of treatments to experimental units allows cause-and-effect conclusions between the explanatory and response variables, because the potential for confounding variables is reduced. If the medicine group's symptom scores are lower than the placebo group's by more than chance would explain (you'll learn how to judge that in Units 3 and 4), you can conclude the medicine caused the lower scores.
Generalizing: depending on the experimental units, it may be unethical or difficult to randomly select them from a population. You can't make randomly chosen people take an experimental drug. In that case, the experimental units are volunteers, and the results apply to the population of units similar to those who participated. Topic 1.10 has the full table of what random selection and random assignment each allow.
Example 3: Stating the conclusion
The 24 volunteers in the allergy experiment were adults with seasonal allergies who answered an ad, and the treatments were randomly assigned. Suppose the medicine group's symptom scores were lower by more than chance would explain. What can the company conclude, and about whom?
Cause and effect: because the treatments were randomly assigned, the company can conclude that the medicine caused lower symptom scores than the placebo.
Generalization: only to adults with seasonal allergies who are similar to these volunteers. The volunteers weren't randomly selected from all allergy sufferers, so the results can't be generalized to everyone with allergies (for example, children or people with year-round allergies).
When you justify a cause-and-effect conclusion on the AP exam, cite the random assignment of treatments. Mentioning blinding or replication doesn't justify cause and effect on its own.
Common mistakes
- Mixing up random assignment and random selection. Random assignment (to treatments) supports cause and effect. Random selection (from a population) supports generalizing.
- Confusing a control group with direct control. A control group is a comparison group. Direct control means keeping other variables the same for all units.
- Describing random assignment vaguely. "Randomly split them into two groups" isn't enough. Say how units are labeled, what random mechanism is used, and how the groups are formed.
- Randomizing across blocks. In a randomized block design, form the blocks first, then randomly assign treatments separately within each block.
- Forgetting to randomize the order in a matched pairs design. If every unit gets the treatments in the same order, time effects are confounded with the treatments.