The real "crisis" in polling

Issue polling is difficult and imprecise. But if anything, the crisis is in political analysis

An old “issue” poll. (Yeah, I think we can do better than this.)

Dear reader,

I joined this week’s 538 politics podcast to discuss the alleged “crisis” in issue polling — a case argued recently by New York Times chief political analyst Nate Cohn. In an article, Cohn writes that “there’s a case that ‘issue’ polling faces a far graver crisis than ‘horse race’ polling” because — as he sees it— pre-election polls in 2022 did not give him issue-based reasons to expect that Democrats were going to do as well as they did in the House or Senate. The issue polls, per this account, failed him. He writes:

But although [2022 horse race] polls performed well, they simply didn’t explain what happened. If anything, the polls were showing the conditions for a Republican win. They showed that voters wanted Republican control of the Senate. They showed that a majority of voters didn’t really care whether a candidate thought Joe Biden won the 2020 election, even though election deniers wound up being clearly punished at the ballot box. Voters said they cared more about the economy than issues like abortion or democracy, and so on.

The Times/Siena polling wasn’t alone in this regard. Virtually all of the major public pollsters told the same basic story, and it’s the opposite of the story that we told after the election. If we judge these poll questions about the issues by the same standard that we judge the main election results — a comparison between the pre-election polls and what we believe to be true after the election, with the benefit of the results — I think we’d have to say this was a complete misfire.

Let me say right off the bat that I do believe that there are problems with issue polling. I even think the steps Cohn later proposes for improving issue polling at the Times will be an improvement in how the press covers public opinion. But on the matter of whether this evidence constitutes a “crisis” in polling, I disagree. In the podcast I detail 4 major objections to Cohn’s argument:

1. Issue polls are non-falsifiable. We do not experience “issue poll” opinions in reality — as we do for “horse race” opinions — and so have little quantitative basis on which to grade their accuracy. Cohn actually concedes this point in the text, but brushes off the caveat to engage in what is really an even more non-falsifiable argument about his own personal poll-informed worldview from 2022. This is not strong footing for the argument.

2. National polls do not meet voters where they are. If you do want to evaluate it the merits of the evidence, consider this: State-level polls in places like Michigan, Pennsylvania, Arizona and Georgia in 2022 revealed a portrait of public opinion that was much closer to the released issue importance of the electorate, according to Cohn. So if we use a different type of “issue” poll, we come to a different conclusion. Surely a “crisis” would be broader than just national issue polls.

3. Is this a crisis of issue polls, or of punditry and political analysis? Whatever indictment Cohn is issuing here is not really against issue polling as a tool, but rather against a particular form of analysis based on particular poll questions about particular opinions about one single recent election. Other polls — about abortion ballot referendums in Michigan, say, or about real-life election deniers on the ballot in — gave us reason to believe that Democrats would be propelled by the issue where it was most salient.¹ Indeed many other analysts came to different conclusions about the election than Cohn did using the same issue polls.

4. “Issue polls” are not set up for this. Because of fundamental limitations on survey research, national toplines cannot give us the granular insights into voter psychology that the national political press asks of them. Other types of polling — such as experiments or “deliberative” polling may do a better job, but they still face shortcomings. There is a long history of political science research into things like “non-attitudes,” response error and question-wording/priming effects, for example. Surely, too, most survey researchers would recoil at the way many journalists paint in broad brushes about politics using singe towlines from so-called “issue” polls. At a certain point, we’re just abusing the tool — using a screwdriver to split wood.²

I want to be clear: None of this is to say that issue polling doesn’t face challenges. Indeed, the landscape of issue polling is particularly fraught with partisan advocacy organizations and biased surveys. But these problems are well understood by high-quality pollsters, even if the press gets things wrong. If we do have worse signals about public opinion now than we did in the past (and I am not convinced that we do!) then the people misusing the tool to craft their narratives bear some of that blame.

. . .

These points echo many themes from my book and the archives of this newsletter, so I thought I’d flag the pod for you. FYI, we do talk about Nikki Haley’s presidential campaign first, so if you want to skip that fast-forward to about the 36-37 minute mark.

Meanwhile, I have been very busy recently with a few projects related to pollster accuracy and transparency, and the fruits of my labor are nearly ready to share. More soon.

Elliott

1. Note that this is what Zaller’s RAS model (1992) would lead us to expect from polling different voters about the same issue across geographies

2. Natalie Jackson, of GQR Research, made a related point in a recent column for the National Journal.