4.5 - Measuring Public Opinion

Elements of Scientific Polling

Sampling requirements

1. Random Sample: everyone in the population has an equal shot at being selected. Letting people opt themselves in makes it non-random.

2. Stratified Sample: the population gets divided into subgroups (race, gender, party, income, age, location, education) and weighted to match real world demographics, without over- or under-sampling any group.

3. Sample Size: around 1,000 people is generally considered scientifically valid.

4. Sampling Error: even a well-run poll has some margin of error since it only surveys a sample, not everyone. A margin of plus or minus 3 percent is typically acceptable, and any legitimate poll reports this.
• Using the margin of error: add and subtract it from each result. If a poll shows Candidate A at 48% and Candidate B at 45% with a margin of error of plus or minus 3 points, A's real support is probably between 45% and 51%, and B's is between 42% and 48%. The ranges overlap, so the race is a statistical tie (within the margin of error), even though A looks ahead.
• Sample size: bigger samples shrink the margin of error, but with diminishing returns. A random sample of about 400 people gives roughly plus or minus 5 points, about 1,000 gives plus or minus 3, and it takes about 4,000 to get down to plus or minus 1.5.
• What it doesn't cover: the margin of error only measures the chance that a random sample is off. It can't fix a biased sample, bad question wording, or people who don't answer honestly.

Question design and framing

1. Neutral Wording: questions have to be clear and unbiased. Leading language or confusing phrasing invalidates results.

2. Neutral Framing: the context around a question can quietly steer answers. For example, asking about drilling in a wildlife refuge right after questions about the environment might push people to oppose it, while asking right after questions about gas prices might push them to support it.

3. Format: whether questions are open-ended, multiple choice, or ranked choice also shapes the results.

Reporting requirements

• What a good poll report includes: who paid for and ran the poll, when it was taken, how many people were asked and who they were (all adults, registered voters, or likely voters), how they were contacted, the exact question wording, and the margin of error.

• Example of overstating: a headline that says "Most Americans now support the plan" when the poll shows 51% support with a margin of error of plus or minus 4 points. Real support could be as low as 47%, so the data doesn't back up "most."

Types of Scientific Polls (Mass Surveys)

1. Opinion Polls: measure public sentiment on a specific issue, like marijuana legalization. Keep in mind opinions shift over time, so old data can go stale.

2. Tracking Polls: repeated over time to chart shifts in opinion, like presidential approval ratings, sometimes surveying the same people repeatedly.

3. Benchmark Polls: usually done by a candidate before officially announcing, to gauge strengths, weaknesses, name recognition, and which groups view them favorably.
• This creates a baseline view of the candidate. Later tracking polls get compared to it to see whether the campaign is gaining or losing ground.

4. Exit Polls: done in person as voters leave polling places, helping predict outcomes and understand which groups voted and why.

Non-Scientific Polling

• Focus Groups: a small group of voters having an in-depth discussion about a candidate or issue. Useful for detailed insight, but not scientifically valid since the sample isn't large or representative.

• Straw polls and opt-in online polls: anyone who wants to can answer, like a poll on a news site or social media. The people who choose to answer usually care more about the issue than most people, so the results don't represent the public.

• Push polls: not real polls at all. A campaign calls voters and asks loaded questions meant to spread negative information about an opponent, like "Would you still support Smith if you knew she raised taxes three times?"