The biennial freak-out over potential error in election polls is now in full force. Republicans, including the president, have gone on something of a news and social media spree recently to argue that the polls are biased against them. Meanwhile, as I showed three weeks ago, technical details about how pollsters are gathering and adjusting their samples might actually be creating polls that are understating Democratic support, a problem few seem ready for.
As I showed in this article and this one, while polls likely have some error in them (this is inherent to how polls work), it is a fool’s errand to try to predict the direction of error ahead of time.
But that does not mean that we are without tools to explore and explain how our polls might be off. Strength In Numbers is doing some pretty sophisticated stuff on this front, since we know that how pollsters adjust their data is now as important as, or potentially more than, how they gather it.
And in this week’s Deep Dive, I want to present one of the new tools I’m using to detect potential bias in the polls ahead of November’s elections.
See, the age-old problem in polling — all polling, not just the Strength In Numbers/Verasight poll — is that the people who will take our polls are often different from those who won’t. In 1936, polls had too many high-income voters. In 2016, too few non-college-educated whites answered their phones.
This problem is called “non-response bias” — since the non-responders are biased compared to the responders. The good news is that pollsters can mostly weight away non-response for different demographic groups by forcing samples to match Census totals (the Census being, of course, a giant poll with a super high response rate!).
The bad news is that weighting can’t fix partisan non-response — supporters of one side quietly opting out of polls at a higher rate than the other, in a way no demographic correction can undo. That’s because there’s no official national tally of Democrats and Republicans to weight to, so pollsters usually find out they had a problem only after the votes are counted.
But what if you could undo bias in a different metric that serves as a proxy for partisan non-response, and that you can detect entirely within the survey data? No more weighting to guesses at what the country’s party ID breakdown is this month!
We know, for example, that our freaky group of survey-takers tends to be super politically and socially engaged. They vote more, volunteer more, donate more to campaigns, and even see more movies in theaters than poll non-responders.
Can we find a way to see if some of our poll respondents are more engaged than the rest? If they are, are they also more Democratic or Republican leaning?
That’s exactly what I did in our August Strength In Numbers poll. I ran an experiment with free-response text questions so I could grade each respondent’s “effort” in answering the poll. This let me compare high- and low-effort respondents and compare their politics.
In other words, in the hypothetical where polls are over-representing high-engagement individuals, can we proxy that with our in-survey measure of survey effort?
It turns out that high-effort respondents are significantly more favorable to one party over the other. And if this measure of effort is at all predictive of the voting behavior of people who don’t answer the polls (who can be thought to have an effort of zero), the polls could be off in terms of vote margin by up to 3.5 points favoring one party this November.
You’ll have to read on to figure out which one. The upcoming midterm elections will serve as an out-of-sample test of this analysis, which has predicted polling errors in recent elections, but only after the fact.