1.12 - Potential Problems with Sampling
Overview
Even a random sample gives a statistic that misses the parameter by a little, just by chance. Bias is a different and bigger problem: a flaw in the method that pushes the statistic in the same direction again and again. This topic covers what bias is, the four named types on the AP exam (voluntary response, undercoverage, nonresponse, and response bias), and how to explain them the way graders want: who is affected and which direction the estimate moves.
What bias means
Bias in a sampling method is a systematic error in the sampling procedure that makes a statistic consistently larger or consistently smaller than the parameter it estimates. Two ideas follow from that definition:
- Bias belongs to the method, not to one sample. A single random sample can miss the parameter by chance; that's sampling variability, not bias. A biased method misses in the same direction over and over.
- A bigger sample doesn't fix bias. It reduces the random scatter, but the statistic still lands in the wrong place.
The figure shows both ideas. Suppose 40% of a town's adults support building a new skate park, so . Each dot is the sample proportion from one simulated sample (40 samples per row, rounded to the nearest 0.01). The random samples center near 0.40. In the voluntary response samples, supporters are more eager to respond, so those samples center near 0.55. Making the voluntary response samples 10 times bigger just packs the dots tighter around the wrong value.
Nonrandom sampling methods
Nonrandom sampling methods, such as convenience samples and voluntary response samples, introduce potential bias because they don't use random chance to select the individuals.
- Convenience sample: the researcher picks whoever is easiest to reach, like the first 50 people who walk into a mall. Those people may differ from the population in ways that matter.
- Voluntary response sample: people choose themselves, like answering an online poll or calling a radio show.
Random selection protects against these problems because chance decides who gets in, not the researcher and not the respondents.
Types of bias
Voluntary response bias
Voluntary response bias may occur when a sample consists entirely of volunteers. People who choose to respond usually care a lot about the topic and have strong opinions, and strong negative opinions are especially likely to show up. Online reviews, call-in polls, and "click to vote" surveys all have this problem.
Undercoverage bias
Undercoverage bias may occur when the sampling method leaves out part of the population, or makes part of the population less likely to be selected. For example, a survey about home internet access that's done entirely online can't reach people without internet, so it will overestimate the proportion of people who have home internet access.
Nonresponse bias
Nonresponse bias may occur when some of the individuals chosen for the sample can't be reached or refuse to respond. It's a problem when the people who respond differ from the people who don't in ways that matter for the study. A random sample with a low response rate can be just as misleading as a voluntary response sample.
Response bias
Response bias may occur when responses to a survey, or measurements of observational units, tend to differ from the true value in one direction. Common causes:
- Question wording bias: confusing or leading questions. "Don't you agree that the dangerous, outdated playground should be replaced?" pushes people toward yes.
- Self-reported responses: people tend to overreport things that sound good (hours studied, minutes exercised) and underreport things that don't (screen time, speeding).
- Sensitive questions: people may not answer honestly about cheating, drug use, or breaking rules, especially when asked face to face.
Telling them apart
Bias | What goes wrong | Question to ask |
Voluntary response | People choose themselves to be in the sample | Did people decide on their own to respond? |
Undercoverage | Part of the population has no chance, or less chance, of being selected | Who could never have been picked? |
Nonresponse | People who were selected don't respond | Who was picked but didn't answer? |
Response | People respond, but their answers are off in one direction | Would people answer this honestly and accurately? |
Undercoverage happens before the sample is chosen: some people were never in the running. Nonresponse happens after: people were chosen but didn't answer. Voluntary response is different from nonresponse because nobody was chosen at all; the respondents picked themselves.
Explaining bias on the AP exam
Naming the type of bias earns little credit by itself. A complete explanation, in context, has three parts:
- Who: which people are left out, overrepresented, or answering inaccurately.
- How they differ: why those people's values of the variable are different from everyone else's.
- Direction: whether the statistic is likely too high (an overestimate) or too low (an underestimate) compared to the parameter.
Example 1: Undercoverage
A school wants to estimate the proportion of its students who skip lunch on a typical day. A staff member surveys a random sample of the students in the cafeteria during lunch. Identify a likely source of bias and its direction.
Who: students who skip lunch are rarely in the cafeteria at lunchtime, so they have little chance of being selected. That's undercoverage.
How they differ: the students who are missed are exactly the ones who skip lunch.
Direction: the sample will have too few lunch-skippers, so the sample proportion will likely underestimate the true proportion of students who skip lunch.
Example 2: Voluntary response
A city council member posts a link on her website: "Tell us: are potholes on your street a problem?" Of the 640 people who responded, 85% said yes. Explain why 85% is probably not a good estimate of the proportion of all city residents who think potholes on their street are a problem.
This is a voluntary response sample, because people decided for themselves whether to respond. Residents who have bad potholes on their street are much more motivated to click and answer than residents whose streets are fine. So the people who responded are more likely to say yes than residents in general, and 85% likely overestimates the true proportion. Getting 640 responses doesn't fix this, because more volunteers just means more of the same kind of respondent.
Example 3: Nonresponse (AP-style)
A researcher mails a survey to a random sample of 500 households in a city, asking how many hours per week the adults in the household work. Only 160 surveys come back.
(a) What is the response rate?
Only 32% of the selected households responded.
(b) Describe how nonresponse could bias the estimate of the mean number of hours worked per week.
Adults who work long hours have less free time to fill out and mail back a survey, so those households are probably less likely to respond. The households that did respond likely work fewer hours, on average, than the ones that didn't. As a result, the sample mean will likely underestimate the true mean hours worked per week for households in the city.
(c) The researcher says, "Next time I'll mail 5,000 surveys so the bias goes away." Is that right?
No. A larger sample reduces sampling variability, but if people who work long hours still respond less often, the estimate will still be too low. To reduce nonresponse bias, the researcher needs a higher response rate, for example by following up with the households that didn't respond.
Example 4: Response bias from question wording
Compare two versions of a survey question:
- Version A: "With college costs rising every year, do you support more state funding for college scholarships?"
- Version B: "Do you support or oppose increasing state funding for college scholarships?"
Version A opens with a reason to say yes, so it's a leading question. It will likely produce a higher percentage in favor than the true level of support, which is response bias from question wording. Version B is neutral: it offers both options and gives no reason for either one.
Common mistakes
- Calling a small sample biased. A small random sample has more sampling variability, but it isn't biased. Bias is a systematic error in one direction.
- Naming the bias without a direction. Say whether the estimate is likely too high or too low, and why, in context.
- Confusing nonresponse with voluntary response. Nonresponse: people were selected but didn't answer. Voluntary response: people selected themselves.
- Confusing undercoverage with nonresponse. Undercoverage: some people could never be selected. Nonresponse: they were selected but didn't respond.
- Thinking a random sample can't be biased. Random selection keeps the researcher's choices out of who gets picked, but nonresponse and response bias can still happen after the sample is chosen.
- Thinking more data fixes bias. A biased method with a huge sample gives a very consistent estimate of the wrong value.