Can we trust impeachment polling?
Some research urges caution, but I see no reason to worry… yet
The takeaway: Pollsters and survey researchers are aware of a phenomenon whereby partisans stop answering polls when the news is bad for their party. But even though impeachment is bad for Trump, I don’t (yet) find evidence that impeachment polling is suffering this fate.
Editor’s note:
Thanks for subscribing! Your membership adds up and makes all this newslettering possible (reminder: I do all this work independently). Please consider sharing online or with a friend; the more readers, the merrier. Remember that apart from getting special articles subscribers can also comment below each post and participate in exclusive threads.
As always, send me your tips about what you’d like to read about next. Or what you don’t want to read. Or your feedback otherwise. Also, cat pictures are nice, so please send them to me! I’m elliott@thecrosstab.com, or @gelliottmorris on Twitter.
Thanks all,
—Elliott
PS! I’m giving away free trials to this newsletter from now until November 1st. I’d love if you shared it with your friends. Click the button below or share this link: https://thecrosstab.substack.com/subfriends
Pollster Mark Blumenthal argued in a piece for FiveThirtyEight last week that big news events can “warp the polls” by suppressing response and completion rates among partisans who are disenchanted by political events. Pollsters call this phenomenon “differential nonresponse bias”. It can be a big issue, but, notably I’m not seeing any evidence of it today.
Blumenthal points us to research of his that shows that survey completion rates—the percentage of people who start and finish a poll—are lower when the news is bad for partisans and higher when it’s good for them. He finds that Trump approvers were more likely to respond to SurveyMonkey polling during confirmation hearings for Brett Kavanaugh, for example, and lower after:
Some people are saying that impeachment might be depressing survey response/completion rates for Trump supporters, but I’m not seeing evidence of this right now.
To tell whether pollsters are being hit by nonresponse bias today, I looked at the percentage of respondents that identified as conservatives in SurveyMonkey’s polling for INSIDER. My theory is: if impeachment was causing Trump voters to not finish their surveys, the number of conservatives in our sample would decrease (because being a Trump voter and a conservative are correlated. But I find no evidence of this. PS: You, too, can get this raw data here.
I might check these data for non-response among Biden voters, too. But for now, know that there doesn’t seem to be any evidence that impeachment is suppressing response/completion rates among Republicans/Trump supporters. You can probably trust the numbers you see.
Still, it’s a good idea to heed Mark’s advice:
And whether you think differential nonresponse bias occurs rarely or often, my advice at the end of yesterday’s article remains pertinent: Caution is always in order when interpreting sudden changes in polling trends. And recontact (panel) studies and voter list samples (that include data on partisanship) are vital in catching non-response bias as it happens.