This article is cross-posted from the Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit neatly into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.

Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.

Upgrade to support Strength In Numbers

It seems like the polls are always wrong these days.

Last night, U.S. House Representative Chris Pappas won the Democratic primary for New Hampshire’s seat in the U.S. Senate by 32 percentage points. But one poll, from the University of New Hampshire, had him winning by just 13 points — a 19-point miss — and another from Democratic polling firm GQR had him up 56 — a 24-point error. While the polls were good on average, that average hides a lot of variance in performance at the level of the individual poll.

And who can forget polls of the Wisconsin primary last month? The final polling average at FiftyPlusOne had Francesca Hong up 20 points, but she ended up losing by about 1. The same night, the final KSTP/SurveyUSA poll of Minnesota’s DFL Senate primary had Democratic House Rep. Angie Craig up 2 over MN Lieutenant Governor Peggy Flanagan, who then won by 19.6. That’s a 22-point miss (and in the opposite ideological direction of the Wisconsin error — a fact that went almost entirely unremarked upon).

In response to these misses, people are again freaking out about the polls. The New York Times called Hong’s loss “stunning.” Tom Nichols of The Atlantic floated the possibility that polls now carry a systematic pro-progressive tilt (based, as far as I can tell, on a sample size of two elections).

Yet there is no real evidence that polls are any worse now than they have ever been. Primary polling has always been particularly fraught; the average Senate or governor primary poll from 2014 to 2024 has missed the final margin between the top two candidates by 13 points.

Instead, what’s happening is that journalists are paying intense attention to primary polls and expecting a level of precision they simply cannot provide. As I argue in my book on polling and democracy, the right response to misses like this isn’t to cast all polls aside, but to lower your expectations for precision.

But don’t mistake this for a claim that the polls are All Right. One thing that has certainly changed in polling recently is that how pollsters process their data matters more than ever. And the recent misfires show how. When I reanalyzed one pollster’s individual-level data from Wisconsin, what I found was not that the polls had too many progressives in them, but that the published toplines were generated using a suboptimal adjustment procedure.

When I re-weighted the same interviews to a voter-file picture of who actually turns out for an August primary, the error on the margin fell from about 24 points in the raw files to about 10 points — a reasonable “error” in a race that saw multiple candidates enter and exit in the final week, and when there was a lot of consolidation against the left-wing candidate in the final days. Most of the pollster’s twenty-plus-point polling error was not in which voters they talked to, but how they counted them.

And that’s not just a primary-polling story. That’s a story about how the entire polling industry has changed over the last decade. The old model of polling — where you randomly dial 1,000 people and make small adjustments to your sample — is dead. Instead, all polls are models now.

This is a post I’ve been working on for a while. I’m sending it out now, as a mid-holiday-week Deep Dive, and preparing some thoughts for tomorrow on the Republican Party’s midterm convention (which they’re calling “Trumpapalooza”) happening September 9–10 in Dallas.

The following premium post is ungated for the public by paying subscribers to Strength In Numbers. Premium members get access to exclusive analysis and a private community for readers on Discord where we discuss election returns, political news, and general politics wonkery through the lens of data.

Upgrade to support Strength In Numbers

Polling has changed from a design-first to a modeling-first problem

You can think of every poll as sitting somewhere on a one-dimensional spectrum. At the far left end of this spectrum, polls are purely “design-based” instruments — where a pollster spends the most time thinking about how they gather a representative sample of the population, rather than how they can adjust their sample to make it representative after gathering the data. In the design-based world, everyone in the population has some non-zero likelihood of answering your survey, which you can equalize with smart design techniques (such as stratification by party registration or geography, etc).

In the pure design world, no adjustment to the sample is needed because the sample is the population in miniature. This model of polling — where people are assumed to be marbles in a bag that you can randomly sample — has been dead for decades. Emil Hurja developed rudimentary adjustment procedures for polls in the 1920s and early 1930s!

At the other end of the spectrum is pure modeling, where a pollster’s raw sample looks nothing like the population and they have to deploy a variety of statistical techniques to force it to make sense. A firm might use some combination of weighting — where individuals who are unrepresented in the sample, based on comparisons to population benchmarks, get more weight in a weighted average of responses, and overrepresented groups get less — and formal modeling (like MRP) to adjust its sample to be “representative.”

At the most extreme point of the model-based world, a pollster collects a convenience sample (maybe through some sort of opt-in survey marketplace, or at their local grocery store) and runs the data through a mess of models and raking algorithms to come up with something that, in backtesting, roughly matches the population on several benchmarks. If you use MRP, you can process unrepresentative survey data using official Census benchmarks for demographics, and rely on other pseudo-official benchmarks for things like past vote and party ID to align the sample politically.

Most polls today exist somewhere along this spectrum from design to modeling. YouGov, for example, pioneered a non-probability sampling technique called “matched sampling“ to conduct its surveys, and uses a very detailed weighting system for calibrating responses to population benchmarks. The Pew Research Center weights its online probability panel by a dozen variables — a heavy, if increasingly typical, lean on modeling — despite also investing hundreds of thousands (millions?) of dollars of R&D on making sure the people invited to the panel are representative of the population at recruitment.

And the New York Times/Siena poll, which had previously heavily invested mostly in the design of its surveys (for example, by sampling respondents based on their predicted party affiliation or registration), recently adopted a state-of-the-art and firmly “model-heavy” weighting procedure called “energy balancing.”

The thing that has happened in polling over the last 20 years is that collapsing response rates have pushed nearly every pollster hard toward the modeling end. But the rise of internet- and text-based survey providers has also helped along the transition. Pew’s telephone response rates fell from 36% in 1997 to 6% by 2018. Peter Enns and Joey Marshall of Verasight put the response rate for even the most rigorous probability surveys at 1.6% to 3.4% today.

When only 1 out of every 100 people are answering your poll, design-based inference can only go so far.

As a result, we are living in a world where all polls are models now. Not 100% modeling, certainly, but at least 50% or 60%. (Those numbers are just guesses; one way to get a real number would be to calculate the error of the NYT/Siena microdata before and after weighting, and then calculate the reduction of error due to “modeling.”)

This shift from design-first to model-first polling is important because it changes how you, the reader, should evaluate pollsters. Because the quality of raw data has decreased so much, when you see a new number fly across your social media feed, it is ever more important to ensure that the person generating that number is an expert adjuster of polling data. You don’t want to end up in a situation where you are trusting data from someone who is literally making up responses for voters based on vibes.

Subscribe now

“Probability” vs “non-probability” doesn’t matter as much when all polls are models

A related transition in the polling industry has been the shift from “probability” methods to “non-probability” ones. This distinction was useful when “probability” was synonymous with “representativeness” — especially compared to “non-probability” polls — but that is no longer true.

For the uninitiated, “probability-based” surveys are those where everyone in the population has some non-zero chance of being interviewed for your poll — e.g., random-digit dialing, or its modern cousin, mailing postcards to people with a survey URL. “Non-probability” surveys are those where some people can never be selected, like an online panel that only includes people who signed up at some website.

In this report for Verasight, Enns and Marshall argue that this distinction can no longer serve as the heuristic for whether a survey is any good. They write that the whole case for probability sampling rests on one assumption: that the people who don’t answer are basically like the people who do. Enns and Marshall’s argument is that this has stopped being true (it basically never was, hence why weighting exists) and that once this assumption is broken, the math that made probability sampling special stops working. (Full disclosure: Verasight conducts the polling for Strength In Numbers.)

Enns and Marshall also make the point that weighting probability samples doesn’t rescue the probability-ness of it all, and it can actually make estimates worse. Their example is to assume less-educated people are underrepresented in your sample, and also assume the ones who did answer are more politically interested than the ones who didn’t. Weighting that group up doesn’t make the missing people magically appear in the sample — it just hands more influence to the unrepresentative respondents who showed up in the sample.

So Enns and Marshall want readers asking two different questions instead: first, what is this firm doing to keep non-response from being systematic — how it reaches people, how many times it tries, how the invitation is worded, what it pays? And second: how have its past surveys held up against known benchmarks, elections being the most visible one of all.

The Verasight crew points to a 2025 analysis by Enns, Amelia Goranson, Jake Rothschild, and Gretchen Streett which found that high-quality non-probability firms actually did better than probability firms in the 2024 presidential election. If you grouped all probability and non-probability separately, probability samples missed the Trump-Harris margin by 3.3 points on average while non-probability samples missed by 4.1.

But if you split those non-probability polls by whether the pollster belongs to AAPOR’s Transparency Initiative or files its data with the Roper Center, the error for transparent pollsters drops to 2.3 points — making them the most accurate group in the entire study, probability pollsters included.

The upshot here is that how a pollster gathered their sample did not separate the good polls from the bad in 2024. Instead, what mattered is the whole of the data-generating process for a survey: how people were recruited, how questions were worded, and how their responses were aggregated to a single statistic.

And the lesson was much the same in 2020, when non-probability pollsters also beat the probability crowd.

In 2018, Pew found that picking the right weighting variables matters far more than picking the right statistical method for coming up with accurate estimates.

But it’s not a model-only world; even the best-performing adjustment procedure, according to Pew, removed only about 30% of the original bias in the sample. The honest read on polling today is that both how you gather and how you process data matters. There’s no longer a “gold-standard” method.

And, frankly, you don’t have to look far to see how obsessing over whether a survey was probability-based can lead you astray. The American National Election Study is a probability poll, run by academics, with a 2020 response rate of 36.7% (media pollsters HATE HIM!). The ANES’s weighted 2020 pre-election estimate still put Joe Biden ahead by 11.2 points (he won by 4.5). That 6.7-point error was the worst in the study’s modern history. When the ANES team went back to find out why, they found “non-ignorable unit non-response” — that Trump supporters were less likely to answer, and less likely to finish the interview if they did, adjusting for all other demographic and geographic factors.

There are other examples. Ann Selzer’s final 2024 Iowa poll had Kamala Harris up 3 in a state Donald Trump carried by 13. The Washington Post‘s 2020 Wisconsin poll had Biden up 17 in a state he won by six-tenths of a point. And somewhere I previously worked, we vetted one probability-based online panel for state-level election polling, and its estimates were routinely too high for us to have any confidence in its ability to forecast future opinion.

Subscribe now

How you weight your data can really impact results

To make this all a little more concrete, here is an example of how much uncertainty in polling is caused by genuine uncertainty about the political or demographic characteristics of the population being sampled.

Our Strength In Numbers/Verasight surveys are weighted to Census demographics and to a three-year running average of party identification from Pew’s NPORS reference survey. Last month I reverse-engineered those weights and re-ran our May through July 2026 waves under different benchmarks for party composition — varying both what year NPORS surveys a pollster used, and how it calculated the Democratic/Republican shares of the electorate.

These decisions moved the Democratic generic ballot average by up to two percentage points. Our published generic ballot across those waves was D+7.1. Updating the party target to the 2026 NPORS vintage moved it to D+7.9. Pushing independent “leaners” into their parties moved it to D+8.7. Pinning the full five-way party breakdown moved it to D+9.2.

That’s two points of margin that are entirely up to our discretion as researchers. Weighting by party ID using any of these methods is a defensible choice. There’s no single “right” way to referee these choices.

Subscribe now

How should pollsters deal with this added uncertainty?

If the results of the polls now hinge so much on how people are counted, then pollsters should update the way they communicate their estimates to the public. They should show not just the topline population percentage for each response, but also the range of potential percentages that could be “right” under different, completely valid weighting procedures. Studies of polls in 2020 and 2024 show that transparency is correlated with accuracy.

Here are a few things I’m trying in my own polling in 2026 and on:

1. Publish more than one number. If a different but equally defensible set of assumptions moves your topline, show that too. Our generic ballot is D+7 one way and D+9 another, and readers deserve to know about this level of uncertainty. I know why pollsters normally don’t do this: It’s a lot of extra work! But you don’t know what the precise point estimate should be, and readers deserve to know that.

2. Sensitivity testing: Check the variables you can’t weight on. Nobody publishes an official count of how many liberals live in America, so pollsters can’t weight by ideology — but you don’t need a target to find out whether your topline is fragile. Just add different variables to your weighting scheme and watch what happens to the results. In Wisconsin, we found that weighting the ideological makeup of respondents to past exit polls moved the August margin of victory for Francesca Hong by about 11 points after demographics and party were already locked down. If somebody had done that analysis ahead of time, they could have communicated about how we didn’t know the ideological makeup of the primary electorate, but that it mattered a lot for accuracy.

3. Simulate non-response. Rebecca Andridge, Brady West and coauthors proposed a procedure for this in 2019: the “measure of unadjusted bias for proportions” (MUBP), which simulates what the “true” population proportion for a survey variable might be depending on the correlation between non-response and certain attitudes. West and Andridge have since shown it works on real elections, testing it against 18 actual pre-election polls. More on applying this to real data in my write-up of the Wisconsin poll miss.

None of this makes polling more accurate, but it does make pollsters more honest about their potential (in)accuracy.

Subscribe now

Getting this wrong has real costs for democracy.

It would be easy to read all of this as an argument that polls are broken and you should ignore them. But that’s actually the opposite of what I’m arguing.

A well-conducted public opinion poll is the only instrument we have that reports what Americans actually want from their government — as opposed to what people on social media are saying, what the loudest faction of a primary electorate wants, etc. The case I made in my book is that polls can represent the public when politicians can’t or won’t.

But that only works if we’re honest about what a poll is — and what a poll is has changed significantly over the last few decades. The old heuristics for quality — design-driven, probability-based — are dead. There is no gold standard anymore.

In this new wild west of survey research, pollsters should use every tool in their toolbox to explain potential error in pre-election estimates, so that the public can “unlock” the use of surveys for other democratic purposes.

Think about what the last year of Donald Trump’s presidency would have been like without the knowledge that nobody likes him and few Americans support his agenda. Polls have been a key part of the fight against the excesses of Trump’s second term as president.

The point, then, about these recent misfires, isn’t that polls are always wrong. It’s that a poll’s topline is the end product of a long chain of decisions — about whom to contact, who answers, whom to count, and how to weight them — that all impact results. As response rates have fallen and sampling has moved online, these “researcher degrees of freedom” have become impossible to ignore.

The polling community has a lot of work to do to explain to the public why it has erred recently, and why those errors do not define the utility of the whole industry. It can start by admitting to itself that the old world of polling is dead; All polls are models now.

Subscribe now

Related Articles

This post is part of The Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.

Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.

Upgrade to support Strength In Numbers

The folks behind the Marquette University Law School poll have responded to my analysis of last week’s polling miss in the Democratic primary for governor of Wisconsin. As a reminder, the final Marquette poll of the race from July 22-27 had Francesca Hong winning 38% of the vote and David Crowley, who ultimately won with 40% of the vote, at 7%. This is a 33-point miss — a “remarkable difference between poll and outcome,” per the director of the poll, Charles Franklin. In margin terms, this is a 31-point miss.

As a second reminder, my analysis looked at the microdata from a non-profit I did some poll methods work for previously called State Navigate, whose August poll had Hong at 44% and Crowley at 20% — a 20-point miss, about 50% less than Marquette.

The explanation from Marquette of their miss is here. Part of their explanation is also that their poll was taken early, before a lot of late consolidation to Crowley. That’s plausible, but also, 31 points is a lot of consolidation. This is one of the hard things about conducting primary polls in general, though; preferences are very fluid and tend to move a lot very late, especially when candidates are entering and exiting the race.

I don’t want to get into a line-by-line back and forth, but I do want to put their opening here. Marquette’s piece begins as such:

G. Elliot Morris argues that the polls in Wisconsin underestimated David Crowley’s support in the 2026 Democratic primary because they included too many young voters (who mostly supported Francesca Hong) and too few older voters (who mostly supported Crowley).

I think a more accurate reading of my article is that the State Navigate survey was primarily biased because they included too many young voters (with a target for the under-30 population ranging from 9% in their August poll to 17% in July. Additionally, in my article I gave three reasons why the polls were off, not just one:

• Most of the miss in polls in Wisconsin is attributable to faulty demographic targets (too many young people). This inflated Hong’s vote margin by somewhere between 5 and 10 points.

• My best guess is that the race moved 6-10 points toward Crowley after pollsters released their final surveys.

• Non-ignorable non-response within demographic categories likely further inflated Hong’s vote margin by 2-5 points.

Fair enough, I did say “in polls” there, though the article was mostly about State Navigate. I think this probably made Marquette think I was attacking them and hurt their feelings, though, which was not my intention. But note I did attribute the lion’s share of the “error” to late movement, as does Marquette’s John Johnson:

I would not describe late movement as leading to “error” in our survey, since we were measuring the electorate as of July 22-27 and not predicting what it would be on August 11.

The remainder of Johnson’s article is a good deep dive into why it’s very hard to know what the age composition of the likely total electorate in Wisconsin is. That’s because the state of Wisconsin does not make age of registrant available for commercial voter file vendors, so you get the “vampire” problem State Navigate termed and which I described in the footnotes of my original article, and in this follow-up.

I’m pretty sure Johnson didn’t read the footnotes or follow-up because I did a sensitivity analysis of the age makeup of the Wisconsin electorate that answers his objections outright. I explained that we didn’t know from the WI file what the unknown % exactly was, that polling data provided a benchmark of about 50% - 75% of the entrants with unknown ages being in the 18-30 camp, and then tested what the analysis would show assuming those percentages. I demonstrated that the poll bias was not linearly related to the age composition; that is, even at a high 18-29 voter share, the weighted State Navigate polling data was still a lot better than what they published in August. I’m sure it would make a big difference vs what they published in July,

I explained, for example, that even if you weight the State Navigate data to a 9% target for ages 18-30, you still get a big Hong overestimation (this data is from August 7-8, the week before the election):

Importantly, I also showed that weighting to a rough guess of the ideological composition of Wisconsin Democratic voters, via 2016 primary exit polls, gave a much more accurate picture of the race (that’s the solid line above).

But none of this answers the big question: What share of the Wisconsin likely primary voter universe is made up of under-30s? We will never know for sure, so we are left guessing — and, importantly, weighting polls to those guesses, which is why sensitivity tests of the kind above are so useful and why pollsters should publish them.

For some informed guesswork, Johnson produces this table of estimates of all registered voters and general election voters using data from the Current Population Survey. It shows that 14% of self-described 2024 general election voters in Wisconsin were ages 18-29.

This appears to be slightly less than the benchmark Marquette weighted to (18% in their published tabs). I think this is because their poll weighted to registered voter benchmarks, and then filtered to the population of likely voters based on their stated intent. (This introduces a lot of noise because people are really bad at estimating how likely they are to vote.) I have asked Johnson on Substack to give me the exact weighting targets and will update this paragraph when he responds.

Unfortunately, I’m pretty sure both of these estimates from Johnson are too high. For one thing, a poll weighted to a lower share is more accurate in predicting the winner, even if you adjust for ideology and for late deciders favoring Crowley. (But maybe that’s just because those weights deflate Hong supporters, but still give an unrepresentative look at the electorate. Fair enough.)

The problem with Johnson’s estimates is that he is looking at a different population than I was. For re-weighting the State Navigate data, I was targeting the population of Wisconsin voters who have cast at least one ballot in a Democratic primary for any office since 2016. Johnson is, in contrast, producing an estimate of the statewide population of likely November presidential general election voters. The latter population is going to have a lot more young people in it, because primary voters skew old.

To demonstrate that point, I have loaded up the L2 voter file for Minnesota, which does not have missing data for voters by age, and is demographically one of the most similar states to Wisconsin. According to L2, the statewide share of registered voters in Minnesota who are between the ages of 18 and 30 is 18%. However, the share of likely Democratic primary voters who are between 18 and 30 is only 11%. That’s a 7-point difference. Additionally, about 14% of validated 2024 voters were 18-30 — 3 points higher than the likely primary electorate.

And according to the Census, Minnesota has about 2 points more voters who are 18-30 than Wisconsin does. (Wisconsin voters are older than Minnesota voters.)

If we take Johnson’s estimate that 14% of Wisconsin 2024 voters were ages 18-30 and subtract 3 + 2 from it, we get 9%.

This is the same target I used in the updated analysis I posted last Thursday, which “controls for the vampires.” In that analysis, I produced the following breakdown of error in the State Navigate data, using the 9% 18-30 target:

1. 10 points having too many young people people in the poll / other weirdness in the State Navigate weights

2. At least 5 points of late movement (5 is what we can detect with the recontact) and plausible up to 10

3. About 5 points of non-response bias

Adjusting for the misspecified age composition gets you within the 13 percentage point average error for the margin between two candidates in primary polls. That would indicate the primary polls and Wisconsin were not any worse than average, after adjusting for what looks like maybe a simple mistake or extra extrapolation from Michigan on the part of State Navigate.

But you can eliminate most of the error in 1 and 3 by weighting your primary poll to a reasonable benchmark for ideological composition. We don’t know what the precise ideological composition of any group of voters is, so see Recommendation 3 in my original analysis. Time for some sensitivity analysis, baby!!

For a few more examples, Strength In Numbers subscriber and the former Secretary of State of the state of Oregon, Phil, calculated the following turnout and young voter percentages in recent primaries in the following states:

• NC - 21% overall turnout, 18-29s 7% share

• OR - 42% overall turnout, 18-29s 6% share

• PA - 22% overall turnout, 19-29s 8% share

• OH - 23% overall turnout, 18-29s 7.5% share

He wrote me:

With 35% overall turnout in WI, my educated guess is that the 18-29 year old share of the D electorate might be closer to 10%, plus or minus 2%.

In summary

1. I’m sorry if I hurt Marquette’s feelings.

2. I already conducted a sensitivity analysis on the youth composition of Wisconsin voters, which I should have just published up top in the first article so I didn’t confuse people, and the conclusions about error do not change.

3. Primary polling is very hard, especially in states where we don’t have good voter file data on age composition. People should adjust their expectations for accuracy downward, especially in polls taken weeks before the actual election (as in the case of Marquette).

4. Marquette’s alternate estimate of the age composition of Wisconsin seems too high; this is because they are not adjusting for the fact that likely primary voters tend to be older.

If I were you, I would just put this on repeat:

Primary polling is very hard, especially in states where we don’t have good voter file data on age composition. People should adjust their expectations for accuracy downward, especially in polls taken weeks before the actual election.

This post is part of The Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.

Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.

Upgrade to support Strength In Numbers

The feedback to my Thursday piece re-analyzing Wisconsin polling microdata has been overwhelming and positive. I have received more emails and texts about this survey than anything I’ve written in a while. I consider this a huge win, not only because I am a millennial/Gen-Z and crave your validation, but because it shows the utility of explaining the ins and outs of polling in public.

I did want to issue a short follow-up post about this. What about the vampires? This post is going in the Data Lab, because it’s for the real sickos out there.

Do not invite the vampires into your poll

The Wisconsin voter file is notorious for having a lot of missing and incorrect data. One of the variables that is particularly errant is the age variable. In their methodology report, for some added context, State Navigate wrote:

“many voters in Wisconsin have a birth year of 1800, and we don’t believe in vampires. These generally are newer voters who skew very young; about half of voters who we polled with a birth year of 1800 say they are in the 18-29 age bracket. Thus, if you were to simply take the voter file’s birth year data, you’ll get a Democratic primary electorate that is majority 65+ and has less than 2% of voters under the age of 30. Absentee requests for the Wisconsin primary (which includes requests for Republican ballots) are 5-6% 18-34. Given that this age group has been notorious for voting late/on Election Day, we find it hard to imagine a Wisconsin Democratic primary electorate that is <8% 18-29. Maybe it is; we don’t know for sure due to the ridiculousness that is the Wisconsin voter file”

Well, I am more than a little wary of using polling data that had too many responses from young people to make any sort of predictions about what age numbers would be on the voter file, so it is another opportunity for sensitivity analysis.

In the plot below, I re-raked the August survey with the under-30 target set by increasingly aggressive assumptions about how old the “vampires” are — from my baseline a priori estimate (unknown ages allocated in proportion to the known ages) up through State Navigate’s estimate (half of unknowns under 30) and past it (70%) — under both the L2C weights and the ideology-free L2B weights.

Notably, stepping up the share of voters who were <30 increases error linearly. Under the L2C weights (which account for ideology and income), the margin goes from Hong +2.4 at my baseline, to Hong +4.9 using State Navigate’s stated floor of an 8%-under-30 electorate, to Hong +6.6 if you assume half of unknowns are under-30, and Hong +8.4 at 70%.

Under the L2B weights (which don’t have ideology in the targets), the same exercise runs from Hong +13.8 up to Hong +19.7, nearly all the way back to the unweighted poll’s +23.8. The reason is simple: the young people who actually answered this survey were overwhelmingly Hong voters (likely too pro-Hong, as the non-response analysis suggests), so every additional young voter you add to the electorate model pushes the poll further from the result.

The adjusted weighted estimates I present — even at a 7-10% under-30 percentage — are still much closer to the result than the original weighted or unweighted numbers. Whatever the true age of Wisconsin’s vampires, the topline takeaways from my original analysis stand: The polls had too many progressives in them, which you can mostly but not entirely solve with weighting — especially if you don’t know the breakdown of the electorate by age! — and a large amount of late movement toward Crowley.

Share

Somebody get the garlic

But about the non-response problem: With the adjusted numbers and new age breaks, the weighted estimates do land farther away from the targets. Does that mean we should conclude there was more non-response among moderates after all?

Well, not entirely. Again, the big problem here is that the Wisconsin voter file can’t produce a good benchmark for age (or other) variables, so pollsters are left flying blind as to what to weight their data to. (The right move here, in my opinion, is to simulate what your poll would show under a bunch of different weights to varying targets, but I will save this for another post.)

To set the stage, here is what the results of the State Navigate poll in August would have been if you allocated undecideds and some minor-candidate voters to Hong and Crowley based on their flows in the July-August recontact study State Navigate conducted. This uses my adjusted weighting model and the maybe-realistic-but-we-don’t-really-know assumption that 50% of people on the Wisconsin voter file with a birth date in the year 1800 are actually under 30 years old (what the polls that had too many young progressives in them indicated).

What is really interesting here is that the base scenario is almost perfect on Hong’s vote share, but underestimates Crowley. That’s likely because there was an additional large swing in his direction after the final August poll wrapped fielding, which polls simply cannot capture. The observed flow to Crowley among supporters of the minor candidates in the recontact study was 1.5:1. I also calculated what the results of the poll would have been if you allocate those voters 2:1 to Crowley vs Hong:

Now, here is the same chart, but if you weight the sample additionally to a benchmark for ideological makeup of the Wisconsin primary electorate. (Remember, the State Navigate poll had a quarter of respondents identifying as Democratic Socialists, which we can be reasonably sure is too high.)

Weighting by ideology recovers about 8 points of vote margin for Crowley.

This means moderates aren’t responding as much to our surveys, right? Well, yes and no. We don’t know if “progressive” is the psychological quality that’s acting on voters to not make them vote. It’s likely something related to one’s ideology: like political engagement. I don’t have the year-vintage breakdown of vote history participation for Wisconsin voters in, say, 2018 (the last time there was a competitive primary in Wisconsin), so I can’t do that math, but maybe someone else could.

But we can use the updated “vampire-adjusted” weights and a reasonable guess at the vote flows in the last week (2:1 to Crowley) and plug into our non-response program (the MUBP — see the original post for an explanation).

The best guess from the authors of the paper I’m basing all this math on is that a reasonable starting point for the impact of non-response is setting phi = 0.5. And you can see why: in the L2Cv weights (which account for ideology), pushing that non-response adjustment up to 100% would have yielded a poll with 20% Hong and 60% Crowley! That’s very bad in the opposite direction of most published polls in Wisconsin.

And to be clear, since the L2Cv weights (solid line) already balance the sample by ideology, the non-response here is something about voting for Hong and Crowley that isn’t proxied by ideological identity in a survey. So you can’t use those trends to say anything about non-response among progressives. That is already taken into account by the level-shift in Hong and Crowley vote at phi = 0.

If, however, you use the L2Bv weights with the non-response dial at the 50% position, you get somewhere around 42% Hong and 40% Crowley. Given they each won about 40% of the vote, that’s a pretty good guess. The residuals here point to about a 5-6 point boost for Hong from non-response.

But again, caution is warranted, since the data for this survey all came from a week before the election. If we had data on election day, the impact of a “non-response” adjustment would be lower. (Technically, I guess, the poll not accounting for future movement in the race is “non-sampling” error, in that it’s error that comes from something other than sampling, but that is different from non-response.)

Share

Polling primaries is very hard

In summary, after adjusting my original analysis for the existence of vampires in Wisconsin, the miss in the original State Navigate polls (+20 points of rHong) breaks down roughly as a result of…

1. 10 points from misspecified age weights

2. At least 5 points of late movement (5 is what we can detect with the recontact

3. About 5 points of non-response

Adjusting for the misspecified age composition gets you within the 13 percentage point average error for the margin between two candidates in primary polls. That would indicate the primary polls and Wisconsin were not any worse than average, after adjusting for what looks like maybe a simple mistake or extra extrapolation from Michigan on the part of State Navigate.

But you can eliminate most of the error in 1 and 3 by weighting your primary poll to a reasonable benchmark for ideological composition. We don’t know what the precise ideological composition of any group of voters is, so see Recommendation 3 in my original analysis. Time for some sensitivity analysis, baby!!

If I were to editorialize a bit, I would say 5 points of ideological non-response in a primary poll is really not bad, all things considered. Especially if it doesn’t impact general election polls or what we know about public opinion writ large.

This post is part of The Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.

Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.

Upgrade to support Strength In Numbers

Following on from last week’s post about the importance of weighting surveys by party identity, a reader asked me whether our new MRP estimates are similarly sensitive to survey weights.

The short answer is no, but this is largely a technicality.

The graph below shows my model’s time-series national estimates of Trump’s approval rating and the U.S. House generic ballot among registered voters since May 2025. The blue line is the published estimates that include Verasight’s survey weights in the regression stage of the MRP model. The red line omits these weights.

These differences are not significant to random variance in the MRP model itself.

So if weighting by party ID matters for surveys, but survey weights don’t impact MRP, what’s going on?

Well, this is largely just a trick I’m playing on you. The MRP model does not weight by party, because it includes a benchmark for the partisan composition of every type of voter in the country, based on their demographics and where they live. The model conditions on those shares to post-stratify by party… which is basically just another way of weighting survey data.

The MRP model I have designed to turn our Strength In Numbers/Verasight polls into state- and district-level estimates of public opinion, explained in detail here, works by first estimating the number of Trump voters, Harris voters, and non-voters as well as the number of Republicans, Democrats, and Independents among every demographic and geographic group in the country. I list these groups separately because they are estimated separately. We have precinct-level results on the former that can be aggregated up to the county level, but we have to estimate the latter.

I estimate that by pushing the raw data for Pew’s NPORS surveys into its own MRP model, which conditions on demographics as well as 2024 past vote, to predict the share of Democrats, Republicans, and independents in each demographic x past vote “cell.” Again, more details in the link above.

The point is that, once we reach the point of running the MRP model on the SIN/Verasight polls, we have already conditioned our projection on party and past vote. We just use “modeling” instead of “weighting.”

But this prompts a few additional questions. First, what if the party ID target used in our post-stratification frame is off? Great question. I take this into account in the post-stratification frame by randomly varying the party ID composition of the electorate in each county by a few percentage points.

Second, what if the Verasight survey were weighted by something not in the MRP model? What if we wanted to estimate not p(vote dem | demographics, geography, party, past vote) but p(vote dem | … + social trust + something else)?

Well, if that something else had no impact on p(vote dem) beyond what could be explained by party and past vote, then we’re in the clear. But figuring that out before an election is, of course, the billion-dollar question. After the 2020 election, YouGov found evidence of political non-response within the party and past vote groups of their panel. This is why communicating uncertainty is really valuable.

In the meantime, the narrow answer to the question we started with is “no, weighting by party doesn’t change the Strength In Numbers MRP estimates.” But that’s because party is in the post-stratification frame.

Thanks for asking!

Share

Subscribe now

My agreement with Verasight (a smart, mixed-mode pollster that does all my polling) is that they get to make all the methodological decisions for our poll while Strength In Numbers drafts question wording and response options. This firewall between editorial and the data-generating process behind the poll is intended to shore up public trust in my reporting; I literally can’t tilt the poll methodology to give me numbers that are more convenient to publish.

But all weighting choices generate a certain amount of uncertainty for polls, and I want to be transparent with readers about how the conventional approach to weighting by party identification could be shaping polling results. This affects poll results for every firm that weights by party ID.

The big question is this: Who counts as a Democrat, Republican, and independent?

There are several ways to answer this question, and each has an impact on a pollster’s results. As we’ll see, Verasight’s data actually fares very well on sensitivity to weighting choices — much better, by comparison, to some other non-probability data collected before the 2024 election.

This post is part of The Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.

Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.

Upgrade to support Strength In Numbers

What is the “right” way to measure party ID?

Pew presents its party ID shares from NPORS in the following graph:

You could count up Democrats, Republicans, and independents in three ways using this data:

1. Count independents who say they “lean” toward one party as belonging to that party, and sum up party shares (47D, 43R, 10I). This is what Pew does.

2. Count independents who say they “lean” toward one party as independents (28R, 29D, 44I).

3. Treat “leaners” as their own group (29D, 19 Lean D, 10 I, 15 Lean R, 28R)

Currently, Verasight weights the Strength In Numbers data using the second methodology listed above, weighting to what I’ll call from now on “3-way party ID” instead of “leaned party ID”. There’s also “5-way party ID” (bullet three above).

So how would our results change if we weighted to leaned or 5-way party ID instead of 3-way? And how do our results change in 2026 if we update the weights from their 2025 to 2026 vintages? (That’s the question I had originally that set off this whole investigation.)

To answer these questions, I reverse-engineered Verasight’s weights for our published Strength In Numbers polling using its weighted demographic and partisan marginals. Then I updated those weights in three separate experimental polls to account for the three new party ID benchmarks. Fake poll A re-weights the published data to 3-way party ID per the 2026 NPORS alone — the same counting rule we use today, just the new vintage. Fake poll B re-weights to leaned party ID per the 2026 NPORS. And fake poll C re-weights to 5-way party ID in the 2026 NPORS. So the gap between the published line and fake poll A is the vintage decision, and the gaps among A, B, and C are the leaner decision.

To account for uncertainty in Pew’s party ID benchmark (their NPORS survey reports a margin of error of 1.8 percentage points in 2026), I repeated each experiment 500 times while randomly varying the party ID results used in the weighting program.

Share

Our polling data, weighted four different ways

The chart below shows the generic congressional ballot among registered voters under each of the three experimental weighting models described above, compared to the average published result from May through July 2026.

Here’s how I interpret this graph:

• Baseline published result is D+7.1

• NPORS shows an aggregate party ID margin of R+0.7 across 2023, 2024, and 2025 — the reference for our published result. Updating the 3-way party ID target alone to D+1 (fake poll A) lifts the generic ballot to D+7.9. The updated party ID targets are a no-brainer, so that’s a free +0.8 points for Democrats.

• Taking independents who lean toward one party and putting them into their partisan group and weighting to that benchmark (47D, 43R, 10I) lifts Democrats’ margin further to +8.7.

• If you balance your sample by 5-way party ID as well (pinning the leaner percent to Pew), the result rises again to D+9.2.

All told, the gap between our published number and the most aggressive party pin is about two points of margin.

For some added context, Trump’s July approval moves from 38%–59% to 37%–61% under the five-way pin, a net swing of two points against him.

To understand why the poll that’s weighted to 5-way party ID is so bullish on Democrats, it helps to understand how Pew’s NPORS party ID breakdown currently differs from the data Verasight sends Strength In Numbers. Under our current weighting approach, the July poll’s three-way split already matches NPORS to the decimal point — with about 30% Democrats, 31% Republicans, and 40% everything else.

But the political makeup of those independents varies a lot between the samples. According to Verasight’s July weighted estimates, 13% of U.S, adults are independents who lean toward Democrats, 10% are Republican leaners, and16% pure independents. Pew, by contrast, has far more leaners on both sides — 19% and 15% — and far fewer pure independents, at just 10%. Here’s a visual comparison:

Weighting to the five-way benchmark upweights both leaner groups by roughly half and cuts the pure-independent share by a third. It also trims the party totals — about three points off the Republican share — because the 2026-only vintage comes along for the ride: a three-year average still carries 2024 and 2025, when Republicans were a bigger slice of adults. But the biggest change is the composition of the middle.

One question you might have, however, is why updating the party ID pin is only worth about ~2points on margin for Democrats, when Pew’s NPORS data shows the nation moving about 4 points left over the last year.

The reason is that the poll is also weighted by recalled 2024 vote, which provides another source of balancing by partisan attributes. So when the new party target pushes weight out of the Republican column and into the independent one, the weighting still has to produce a sample that voted Trump+1 in 2024. It makes up the difference by upweighting the Trump voters inside the newly enlarged independent bloc. The party pin gives; the 2024-vote pin takes most of it back.

We can prove it by deleting the 2024-vote pin and re-running fake poll A. Without that anchor, updating the party target to 2026 moves the pooled margin from D+7.1 to D+10.0 — a shift of nearly three points, right in line with the naive arithmetic. But it gets there by drifting the sample’s recalled 2024 vote from about Trump+1 to Harris+2, a 2024 electorate that technically never existed. That’s exactly the failure the vote pin exists to prevent. After the weighting model weights by both party ID and past vote, it’s left drifting up “just” to D+9.

However, the statistical cost is of all this weighting is real, if modest: the design effect of the weights (a measure of added variance from the underlying balancing model) rises from about 1.04 to about 1.15 in a typical wave, which trims the effective sample size of a 1,500-person poll by roughly 125 respondents — or, a widening of uncertainty intervals by about five percent.

And there’s the additional uncertainty from balancing by multiple groups simultaneously, as in the 5-way party ID pin. NPORS is a survey with its own margin of error; a proper margin of error for our survey should take into account that Pew’s estimate of 19% for “lean Democrat” really means somewhere between about 17% and 21% — and you can repeat that exercise for every slice of the target we’re weighting to.

As mentioned above, I account for this by randomly varying the party ID target I weight our Verasight data to by the amounts implied by Pew’s sample sizes for each party ID group. Each of 500 alternative NPORS distributions is fed into a computer program that re-weights the pooled poll to give a final margin of error that combines uncertainty from both surveys. This contributes a further 0.9 points of standard error to the five-way margin, about 0.7 to the leaned margin, and roughly 0.5 to either 3-way margin (which pin fewer categories with larger sample sizes). Adding the two in quadrature and doubling for a 95% interval gives a total margin of error of about ±2.9 points on the published and 3-way 2026 margins, ±3.2 on the leaned margin, and ±3.4 on the five-way margin.

The upshot is the more precise you are trying to be with your party ID weighting scheme, the more potential statistical error you are adding into your poll.

Does weighting by party ID make a poll more accurate? (Yes.)

Unfortunately without a crystal ball, we cannot know which weighting scheme will make our poll more accurate for 2026. But we can see which approach would have been most accurate in past elections.

To figure this out, I took the raw data from the 60,000-person Cooperative Election Study surveys from 2024 and 2020 and reweighted them the same way I described above: Each year’s poll was weighted to be representative by Census demographics, then given a weight according to its party ID in three ways — party pinned 3-way, leaned 3-way, and 5-way, using the NPORS distribution from each cycle.

The results are shown in the graphic below.

In deciphering the above, start with the bottom panel. Trump won 50.7% of the two-party vote, but when weighted to demographics alone, the CES gave him just 43.2%. That’s a whopping 7-point bias in vote share, which would make for one of the worst national pre-election poll misses since 1952.

But weight by party, the picture changes dramatically.

The 3-way party weight recovered most of the gap between the result and original estimate, putting Trump at 48.4% of the two-party vote. The leaned and five-way pins got within a point — 49.9% and 50.0% — and on post-election recalled vote (not shown) both were essentially exact.

Weighting by party was less efficacious in 2020, and unlike in 2024, weighting by leaned party was not additionally helpful in reducing bias; nothing fully fixed that year’s Democratic lean. Weighting by demographics alone put Trump at 41.9% against an actual 47.7%. The three-way party weight got him up to 45.2%, and the leaner pins added just two tenths of a point.

The conclusion from the historical data is that weighting by party is a must, but weighting by leaned party is not a proven solution to reduce bias. The only thing we know for a fact the extra weighting does is increase sample design effects and thus uncertainty. It makes sense that some pollsters would not want to make that tradeoff (especially if they were designing their methodologies before 2024!).

For good measure, I ran one final test on our own Strength In Numbers/Verasight July sample: I held out the 2024-vote benchmark from the weighting model and calibrated the survey by party in the three methods above. Those results are shown below:

When weighted just to demographics, Verasight’s sample is off (though meaningfully less biased than the CES) when predicting the results of the 2024 election. The conclusion from this is that weighting by some political variable is necessary. Weighting by unleaned 3-way party ID does not reduce much bias, suggesting that the independents in the sample lean Democratic by past vote. The five-way party break with pre-election 2024 NPORS numbers recovered the result of the election almost perfectly, while the leaned three-way was off by less than a point.

Still, note that all weighting specifications came within the margin of error of 2024 recalled vote, and of each other.

Share

Weighting by party is not black and white

So why not switch tomorrow? A few considerations.

First, pinning the leaners only makes the poll more accurate if the leaner benchmark is right. Unfortunately, while NPORS is a high-response rate survey developed by some of the smartest minds in polling, it is still only a survey. That means it’s subject to the same sampling and non-sampling error every other poll is. If the leaned party ID is off by 2 points, all polls that weight to it will also be off. Verasight uses a three-year running average of three-way party ID to minimize potential error in the benchmark, which I think is a sensible cautious approach.

And second, NPORS only comes out once a year. Even a perfectly calibrated current target gets stale over time. The 2026 NPORS was fielded from January to June 2026. By Election Day, the entire sample will be 5-months out of date, and the oldest interviews will be nearly a year old! That’s a lot of stale data exactly when polling accuracy matters most. The three-way mix is less likely to go wildly out of whack than the 5-way.

This analysis shows that weighting by some benchmark for party is a good idea. We have done that. The second-order questions (3-way leaned or 5-way split? what about 7-way?) are less important and unresolved.

Finally, there’s the editorial factor I started with: Verasight develops the methodology for the Strength In Numbers poll, not me. If they decide it’s best to update their approach, I’ll follow their lead.

Subscribe now

Got a weighting or methods question you want the Data Lab to dig into? Leave it in the comments below!

Related Articles

This is the inaugural post for the Strength In Numbers Data Lab, a new section of this publication for sharing extra charts, deep polling nerdery, and writing about methods work etc. for the devoted reader of this newsletter. If you download the crosstabs of our monthly Strength In Numbers/Verasight poll, this one’s for you.

The idea for this section is simply to publish more, less polished, and more technical content, both because I know there are many people who are hungry for more content and because there’s stuff I want to keep off the full email list. See, a lot of the analysis I do behind this scenes never makes it into a proper article suited for 70,000+ readers, whether that’s because of the editorial calendar, it’s just one chart, or a reader asks a question that deserves a real answer but is too niche for most subscribers. I tend to file these extras away or send them off as emails where they die in private. That’s a big no-no from a blogging point of view (Always Be Posting) so from now on I’m going to be sharing that work here, outside the Strength In Numbers home vertical (and your inbox) but still public for the real sickos.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here.

Like the main site, most Data Lab posts will be free, but I’ll reserve the occasional longer article for paying subscribers as a bonus for making all of this possible. Upgrade your subscription for full access.

Upgrade to support Strength In Numbers

Finally, as the staff here at Strength In Numbers continues to grow, I’m hoping the Data Lab will also be home to contributed musings from our survey research assistants and other polling professionals mining SIN data for insights. If that’s the type of thing that interests you, email me.

Short programming note: I’m coming back from paternity leave in two weeks, on July 27, when Strength In Numbers will return to full editorial strength and I’ll have the results of our new July poll to share. David and I return for our first summer podcast/Substack live show that Thursday, July 30. Get signed up so you don’t miss out!

With all that out of the way, let’s dive in:

What’s up with people who disapprove of Trump but don’t vote Democratic?

After my May piece on cross-pressured voters, a reader named Laurence asked on the SIN Discord for me to run some special numbers on our polling microdata. That piece was written in response to some bad punditry about Democrats’ midterm polling — the broader point I was making was that you shouldn’t expect everyone who disapproves of Trump to vote for Democrats for Congress, and grading them against that expectation was unfair/misleading

Laurance wanted me to take the Trump disapprovers who still aren’t voting Democratic and carry at least one Republican “anchor” — a 2024 Trump ballot, Republican party ID or lean, or conservative self-ID — and split them up by which anchors they carry. How does each combination vote on the generic ballot? In other words, don’t just count the anchors — ask which ones actually hold.

I recommend you read the piece linked above to get a better idea of how this works, but basically what we’re doing is looking at the subset of people in our poll who say they disapprove of Trump but aren’t voting Democrat and trying to figure out why — and, importantly, how many of those voters Democrats might actually win.

To answer Laurence’s question, I stacked all six of our Strength In Numbers/Verasight monthly polls from January through June 2026 into one big mega poll. This gives us 8,818 registered voters and enough sample to slice the disapprovers thinly.

And here’s where the microdata is really powerful (and why I think what we’re doing here at SIN is so cool). If you sort voters by Trump approval and generic-ballot vote and then count the GOP anchors in each cell, we find that among disapprovers who are undecided, 40% carry one or more GOP anchors. Among disapprovers who say they’ll vote GOP, 90% have Republican anchors. That’s the group I called “closet Republicans” in my May piece. There are a lot of disapproves who look more like Trump supporters/Republicans than Democratic voters, all else being equal.

Now to Laurence’s actual question: here’s the generic ballot for each anchor combination, split by Trump approval:

This chart is great, even if it takes a while to grasp, because now we can see how your anchors condition your behavioral response to Trump’s presidency. Disapproving of Trump when you’re otherwise a solid Trump voter doesn’t mean you’ll become a Democratic voter in the midterms.

Here are the numbers: Voters carrying all three anchors (a Trump ‘24 ballot, Republican in party affiliation, and conservative self-ID) are still voting Republican 76% to 16, so disapproving of Trump barely dents them. Voters who pulled the lever for Trump in ‘24 and call themselves Republicans but not conservatives stay Republican too, at 63% to 26.

But drop the party label and everything changes. Disapprovers who voted for Trump in 2024 yet call themselves neither Republican nor conservative now go Democratic 62% to 22, an outright flip. The Trump-plus-conservative-but-not-Republican cell (second column, second row) points the same direction, at 50% Democratic to 29 Republican (though it contains just 24 respondents across six months of polling, so treat that finding as directional).

The pattern is hard to miss. The subgroups that keep “Republican” in their identity stay Republican by 37 points or more in the generic ballot even while disapproving of the man leading their party, — while both subgroups without it defect to the Democrats by a margin of 30-plus points. Party identity is doing a lot of work, even among Trump voters.

This is why I called this group “closet Republicans,” and I think that characterization stands.

Share

The usual caveats apply: this analysis pools six monthly waves of data, so some of the underlying data may be out of date, and some of these breakdowns have only a few dozen voters in them so I wouldn’t take the exact estimates to the bank. But I think the broader point about GOP identity stands — and, more to the point of the Data Lab, I think I’ve answered Laurence’s question now!

If you have a question you want me to answer with our poll, leave it in the comments below or on the SIN Discord and it might become a future dispatch.

Subscribe now

Related Articles

Welcome to the Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit neatly into the normal editorial vertical of the full site/doesn’t get the full editing treatment before publication. If you’re familiar with laboratory science, the idea is basically to open-source the lab notes for all the polling I run and explore stuff like survey experiments in more detail. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

The Data Lab will also be a place for general miscellany that committed readers will be interested in, but might marginalize the wider audience. I also cut a lot of charts from the full newsletter that I’d like to share somehow, and have some insights into running a Bayesian model-driven polling operation and independent media company that I think folks will find interesting. Finally, as the staff here at Strength In Numbers continues to grow, I’m hoping the Data Lab will also be home to contributed musings from a network of survey research assistants and other polling professionals mining SIN data for insights. If that’s the type of thing that interests you, email me.

In terms of content mix, the goal is that well above half of Data Lab posts will be free. Occasional deeper dives will be paywalled for members, whose support makes all of this possible. Upgrade your subscription for full access:

Upgrade to support Strength In Numbers

The Data Lab is intentionally opt-in for the true Strength In Numbers sickos, so if this at all sounds interesting to you, go to your settings for this publication and opt in to emails from the Data Lab. You should see a menu like the one in the screenshot below with a button to click to enroll in the new vertical. (And you might as well sign up for podcast notifications while you’re at it!)

The first Data Lab post is coming this week. If you have a suggestions for something else I should write, or a question about my polling, leave that in the comments below! Comments are unlocked for all readers.

Leave a comment

We (Elliott + Verasight, my survey partner!) just went into the field with our January poll, and part of the process for that survey was compiling all the questions y’all sent in so we could pick a few to run. For transparency (and fun!), I figured I’d share the complete list of your questions below. I have done a very rough edit of the question text and response options for clarity. Not all of these questions will be used this year, and most are not in the final form we would use in a poll; I’m just sharing them for fun.

Happy reading. If these spark any additional questions or comments, drop ‘em below!

Share

Leave a comment

Read more

I have activated a special coupon for the 2025 holiday season. Click the button below and get 20% off an annual subscription to Strength In Numbers, no questions asked. Perfect for new subscribers, monthly subs looking to save some money annually, or as a gift to a loved one!

Get 20% off for 1 year

Today I have a short, interactive post for the whole audience.

One of the things that sets this Substack apart from other political analysis blogs is that Strength In Numbers sponsors its own polling, conducted in partnership with a mixed-mode polling firm called Verasight (“mixed mode” means they source interviews via multiple modes: in this case, via the internet, text messages, and phone calls). And we (Elliott + Verasight) also let subscribers to the site suggest their own questions for us to ask 1,500-2,000 U.S adults each month.

As far as I know, SIN was the first major politics publication on Substack to make bespoke, audience-driven polling a regular part of editorial strategy. This turned out to be one of the best ideas I had in 2025; the polls generate a lot of back-and-forth with readers (I try to respond to every one of you!) and drive press coverage for the website. The idea was so good that others have followed in our footsteps! Yay for independent polling!

These monthly surveys generate a lot of insights beyond the normal toplines and crosstabs you see from many polls. Because I am a pollster and a statistician/data scientist/computational social scientist at heart, I’ve been able to use the raw data Verasight gives me to do a lot of cool things, including visual analysis of open-ended text responses and estimation of public opinion at the state level (and, soon, maybe even counties and congressional districts). Of course, there’s also everyone’s favorite feature of the monthly poll: line charts.

So far, I have devoted about half of each survey to a core set of tracking questions I ask again and again each month. Then, the rest of the survey is filled in with rotating trackers when they become relevant (tariffs, for example, have been in and out of the news since January), plus one-off questions submitted by subscribers.

And this is where you come in.

What questions do you want us to ask in 2026?

I want everyone who reads this newsletter to suggest one question you’d like us to track on a regular monthly basis in 2026. These questions should be suited to trending (ie, visualizing on line charts) and relevant over the course of the whole election cycle (January - November; we’re going to take December 2026 off). The comments are open below for your suggestions.

The questions I’ve been regularly asking so far are:

• Self-reported likelihood to vote in midterms

• U.S. House generic congressional ballot

• Most Important Problem for 11 issues:

  • Jobs and the economy

  • Prices/inflation

  • Trade with other countries

  • Foreign policy

  • Immigration

  • Border security

  • Deportations

  • Education

  • Health care

  • Government funding and social programs

  • Crime and public safety

• Party and political figure feeling thermometers

• “Which party do you trust most to handle ___” for each of the aboved 12 issues

• Trump approval

• Trump approval for those 12 issues

• Right direction/wrong track

And I plan to add the following questions to the regular rotation:

• Do you think your personal financial situation will get better or worse in the next year? (Better / worse / stay the same / don’t know)

• Two open-ended questions asking what single issue Trump is doing best/worst on. (Used to see if I should add or subtract an issue from the issue tracking.)

• What news story did you hear the most about over the last month?

My goal next year is to somewhat dramatically expand the data products produced by the data in the monthly poll, including making regularly updated custom graphics from the interviews we conduct. That would include crosstabs, automated graphs for trended questions, analysis of open-ended text, and MRP modeling for issue questions. We are also planning on sharing the data with academic data archives, and maybe even open-sourcing some of the analysis pipelines on GitHub (stay tuned).

Full disclosure: Some of these features will probably get locked behind a paywall at some point to help grow the business, but I intend to keep the vast majority of insights from the polls freely accessible to the public.

So, please fire away! I want to know what questions you are curious about, so I can provide an interesting and actionable survey to readers. The whole reason we started this project was to make polling more transparent and relevant to the average American.

When you’re writing your question, keep these guidelines in mind:

• Please only submit one question each. I promise to read every submission, but that’s going to be a lot of work already (SIN is closing in on 50,000 readers), so please don’t abuse the comments.

• Make it one concept per question. Avoid “and/or” mashups; they’re harder to interpret and trend cleanly

• Try to use neutral wording. It is important that questions do not lead respondents to answer a certain way. We’ll lightly edit for clarity/neutrality and keep wording stable over time for trend comparability.

• Include response options if you can. Best formats for monthly tracking are strongly/somewhat approve/disapprove, better/worse/neutral, right/wrong track, favorability scales, 4-point “a lot/some/not much/none”, and scales of numeric integers.

Leave your questions below — or, if you’re a member of the Strength In Numbers Discord, tag me there.

Elliott

Leave a comment

Share