How you word a poll question can have a large impact on results
Our new poll finds question wording can swing opinion by 20-60 points on immigration, the budget, and transgender rights. That's a problem for people who use polls to tell politicians what voters want
Seventy-six percent of Americans support a “path to citizenship” for undocumented immigrants, according to a new Strength In Numbers survey out today. But only 45% support “amnesty” for essentially the same people.
A majority in our poll also opposes “gender-affirming care for minors,” but a plurality supports “parental rights to pursue doctor-recommended treatment” for kids diagnosed with gender dysphoria. And a supermajority of voters say they would like to reduce the national debt, but cuts to Medicare and infrastructure spending are deeply unpopular.
Many Americans hold notably contradictory beliefs on a wide spectrum of policy domains.
The popular view of voters — ideological, easily sorted into left and right camps, easy to predict how they’ll vote based on policy preferences — is wrong. As I have written, many voters don’t fall neatly onto a single left-right ideological spectrum (other research has affirmed my work on this). And there is a non-ideological, “anti-system” dimension to U.S. politics that at least rivals, and in some cases may even surpass, the predictive power of left-right ideology.
To this pile of caveats to such phenomena as the median voter theorem, “popularism,” and political strategy writ large, let me introduce another complication: the “median voter” might not exist. Or more precisely, the median voter is in two (or more places) at once, depending on how you ask them a question. The median voter is like Schrödinger’s cat — both liberal and conservative until a question forces them to pick an answer.
I know that sounds like pollster navel-gazing, but I think you will hear me out because the numbers here are wild. About 64% of U.S. adults give contradictory responses to questions about transgender rights, according to the individual-level data in our poll. On immigration, 51% contradict themselves internally. Somewhere between “many” and “most” Americans hold contradictory beliefs both across and within policy domains.
The median voter is neither liberal nor conservative, but both — and neither — depending on which words you put in front of them. The average American is superposed — carrying competing instincts that collapse into a position only when a specific question forces them to choose.
Jerusalem Demsas recently touched on this problem in a piece called “Against Thoughtless Moderation,” in which she responds to backlash over a poll on transgender rights published by pro-liberalism magazine The Argument (Demsas is the editor of the outlet). Her argument was that politicians shouldn’t just chase poll numbers with policies — they should engage substantively with what voters actually think.
When it comes to being “against thoughtless moderation,” Strength In Numbers wholeheartedly agrees with Demsas (nuance in polling is kinda my thing!). But I want to push the argument a step further — because the more you look at public opinion data, the less clear it becomes what voters actually think. Consider that The Argument’s own poll found the public simultaneously supports non-discrimination protections for transgender Americans and Republican-backed laws that would require them to use the bathroom of their birth sex.
But when poll findings contradict each other like this, which result do you follow? “Go to where the median voter is” is a simple enough strategy — if you know where the median voter is. But the closer you look at the data, the harder it becomes to believe anyone who repeatedly claims to know what the median American always wants. The problem isn’t just that politicians read polls too simplistically. It’s that the polls themselves, depending on how you word a question, can point in completely opposite directions.
This week’s Deep Dive: Many Americans hold contradictory opinions on the same policies. This makes figuring out what the public wants genuinely hard — a product of both science and art — and poll-seeking political strategy particularly fraught. This is a big problem for everyone (including me!) who wants polls to positively influence the political process as a proxy for the voice of the people.
1. Question wording choices can swing opinion on immigration and trans rights by up to 60 points
In this survey, I asked each respondent between one and two versions of the same broad policy question on immigration, preferences for federal spending and cuts, and transgender rights. Each policy contrast was separated by unrelated items so respondents wouldn’t notice the pairing. (See the full topline document at the end of this piece for more.)
One question in the poll, for example, asked our sample of 1,035 adults whether they supported “an amnesty program that would grant legal status to people who entered the country illegally.” The second asked about “providing a path to citizenship for undocumented immigrants who have resided in the United States for at least ten years, paid taxes, and passed a background check.”
The “amnesty” question split respondents almost exactly down the middle, with 47% of U.S. adults opposed and 45% in favor. That’s net R+2 (shorthand for a 2-point Republican-leaning advantage). But the “path to citizenship” frame produced a landslide — 76% in favor, 18% opposed, net D+58. That’s a 60-point swing driven entirely by question wording.
I also asked whether adults would support deporting undocumented immigrants who had lived in the U.S. for a long time but had American-citizen children. Adults said they’d favor letting these residents stay by a 14-point margin (D+14).
These results are visualized in the plot below, in green, alongside the other questions I asked on transgender rights and spending policy. The amnesty question leans Republican; the citizenship and deportation questions lean Democratic:
The full results and analysis below are for paid subscribers. The kind of original polling analysis and experimentation I’m publishing here at Strength In Numbers doesn’t exist anywhere else in the media today. And survey data isn’t cheap! Paid subscribers make it possible for me to ask these questions on custom polls, and to spend my time working on public-interest journalism full-time. Subscribe here.
