This post is part of The Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.

Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.

Upgrade to support Strength In Numbers

The folks behind the Marquette University Law School poll have responded to my analysis of last week’s polling miss in the Democratic primary for governor of Wisconsin. As a reminder, the final Marquette poll of the race from July 22-27 had Francesca Hong winning 38% of the vote and David Crowley, who ultimately won with 40% of the vote, at 7%. This is a 33-point miss — a “remarkable difference between poll and outcome,” per the director of the poll, Charles Franklin. In margin terms, this is a 31-point miss.

As a second reminder, my analysis looked at the microdata from a non-profit I did some poll methods work for previously called State Navigate, whose August poll had Hong at 44% and Crowley at 20% — a 20-point miss, about 50% less than Marquette.

The explanation from Marquette of their miss is here. Part of their explanation is also that their poll was taken early, before a lot of late consolidation to Crowley. That’s plausible, but also, 31 points is a lot of consolidation. This is one of the hard things about conducting primary polls in general, though; preferences are very fluid and tend to move a lot very late, especially when candidates are entering and exiting the race.

I don’t want to get into a line-by-line back and forth, but I do want to put their opening here. Marquette’s piece begins as such:

G. Elliot Morris argues that the polls in Wisconsin underestimated David Crowley’s support in the 2026 Democratic primary because they included too many young voters (who mostly supported Francesca Hong) and too few older voters (who mostly supported Crowley).

I think a more accurate reading of my article is that the State Navigate survey was primarily biased because they included too many young voters (with a target for the under-30 population ranging from 9% in their August poll to 17% in July. Additionally, in my article I gave three reasons why the polls were off, not just one:

• Most of the miss in polls in Wisconsin is attributable to faulty demographic targets (too many young people). This inflated Hong’s vote margin by somewhere between 5 and 10 points.

• My best guess is that the race moved 6-10 points toward Crowley after pollsters released their final surveys.

• Non-ignorable non-response within demographic categories likely further inflated Hong’s vote margin by 2-5 points.

Fair enough, I did say “in polls” there, though the article was mostly about State Navigate. I think this probably made Marquette think I was attacking them and hurt their feelings, though, which was not my intention. But note I did attribute the lion’s share of the “error” to late movement, as does Marquette’s John Johnson:

I would not describe late movement as leading to “error” in our survey, since we were measuring the electorate as of July 22-27 and not predicting what it would be on August 11.

The remainder of Johnson’s article is a good deep dive into why it’s very hard to know what the age composition of the likely total electorate in Wisconsin is. That’s because the state of Wisconsin does not make age of registrant available for commercial voter file vendors, so you get the “vampire” problem State Navigate termed and which I described in the footnotes of my original article, and in this follow-up.

I’m pretty sure Johnson didn’t read the footnotes or follow-up because I did a sensitivity analysis of the age makeup of the Wisconsin electorate that answers his objections outright. I explained that we didn’t know from the WI file what the unknown % exactly was, that polling data provided a benchmark of about 50% - 75% of the entrants with unknown ages being in the 18-30 camp, and then tested what the analysis would show assuming those percentages. I demonstrated that the poll bias was not linearly related to the age composition; that is, even at a high 18-29 voter share, the weighted State Navigate polling data was still a lot better than what they published in August. I’m sure it would make a big difference vs what they published in July,

I explained, for example, that even if you weight the State Navigate data to a 9% target for ages 18-30, you still get a big Hong overestimation (this data is from August 7-8, the week before the election):

Importantly, I also showed that weighting to a rough guess of the ideological composition of Wisconsin Democratic voters, via 2016 primary exit polls, gave a much more accurate picture of the race (that’s the solid line above).

But none of this answers the big question: What share of the Wisconsin likely primary voter universe is made up of under-30s? We will never know for sure, so we are left guessing — and, importantly, weighting polls to those guesses, which is why sensitivity tests of the kind above are so useful and why pollsters should publish them.

For some informed guesswork, Johnson produces this table of estimates of all registered voters and general election voters using data from the Current Population Survey. It shows that 14% of self-described 2024 general election voters in Wisconsin were ages 18-29.

This appears to be slightly less than the benchmark Marquette weighted to (18% in their published tabs). I think this is because their poll weighted to registered voter benchmarks, and then filtered to the population of likely voters based on their stated intent. (This introduces a lot of noise because people are really bad at estimating how likely they are to vote.) I have asked Johnson on Substack to give me the exact weighting targets and will update this paragraph when he responds.

Unfortunately, I’m pretty sure both of these estimates from Johnson are too high. For one thing, a poll weighted to a lower share is more accurate in predicting the winner, even if you adjust for ideology and for late deciders favoring Crowley. (But maybe that’s just because those weights deflate Hong supporters, but still give an unrepresentative look at the electorate. Fair enough.)

The problem with Johnson’s estimates is that he is looking at a different population than I was. For re-weighting the State Navigate data, I was targeting the population of Wisconsin voters who have cast at least one ballot in a Democratic primary for any office since 2016. Johnson is, in contrast, producing an estimate of the statewide population of likely November presidential general election voters. The latter population is going to have a lot more young people in it, because primary voters skew old.

To demonstrate that point, I have loaded up the L2 voter file for Minnesota, which does not have missing data for voters by age, and is demographically one of the most similar states to Wisconsin. According to L2, the statewide share of registered voters in Minnesota who are between the ages of 18 and 30 is 18%. However, the share of likely Democratic primary voters who are between 18 and 30 is only 11%. That’s a 7-point difference. Additionally, about 14% of validated 2024 voters were 18-30 — 3 points higher than the likely primary electorate.

And according to the Census, Minnesota has about 2 points more voters who are 18-30 than Wisconsin does. (Wisconsin voters are older than Minnesota voters.)

If we take Johnson’s estimate that 14% of Wisconsin 2024 voters were ages 18-30 and subtract 3 + 2 from it, we get 9%.

This is the same target I used in the updated analysis I posted last Thursday, which “controls for the vampires.” In that analysis, I produced the following breakdown of error in the State Navigate data, using the 9% 18-30 target:

1. 10 points having too many young people people in the poll / other weirdness in the State Navigate weights

2. At least 5 points of late movement (5 is what we can detect with the recontact) and plausible up to 10

3. About 5 points of non-response bias

Adjusting for the misspecified age composition gets you within the 13 percentage point average error for the margin between two candidates in primary polls. That would indicate the primary polls and Wisconsin were not any worse than average, after adjusting for what looks like maybe a simple mistake or extra extrapolation from Michigan on the part of State Navigate.

But you can eliminate most of the error in 1 and 3 by weighting your primary poll to a reasonable benchmark for ideological composition. We don’t know what the precise ideological composition of any group of voters is, so see Recommendation 3 in my original analysis. Time for some sensitivity analysis, baby!!

For a few more examples, Strength In Numbers subscriber and the former Secretary of State of the state of Oregon, Phil, calculated the following turnout and young voter percentages in recent primaries in the following states:

• NC - 21% overall turnout, 18-29s 7% share

• OR - 42% overall turnout, 18-29s 6% share

• PA - 22% overall turnout, 19-29s 8% share

• OH - 23% overall turnout, 18-29s 7.5% share

He wrote me:

With 35% overall turnout in WI, my educated guess is that the 18-29 year old share of the D electorate might be closer to 10%, plus or minus 2%.

In summary

1. I’m sorry if I hurt Marquette’s feelings.

2. I already conducted a sensitivity analysis on the youth composition of Wisconsin voters, which I should have just published up top in the first article so I didn’t confuse people, and the conclusions about error do not change.

3. Primary polling is very hard, especially in states where we don’t have good voter file data on age composition. People should adjust their expectations for accuracy downward, especially in polls taken weeks before the actual election (as in the case of Marquette).

4. Marquette’s alternate estimate of the age composition of Wisconsin seems too high; this is because they are not adjusting for the fact that likely primary voters tend to be older.

If I were you, I would just put this on repeat:

Primary polling is very hard, especially in states where we don’t have good voter file data on age composition. People should adjust their expectations for accuracy downward, especially in polls taken weeks before the actual election.

This post is part of The Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.

Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.

Upgrade to support Strength In Numbers

The feedback to my Thursday piece re-analyzing Wisconsin polling microdata has been overwhelming and positive. I have received more emails and texts about this survey than anything I’ve written in a while. I consider this a huge win, not only because I am a millennial/Gen-Z and crave your validation, but because it shows the utility of explaining the ins and outs of polling in public.

I did want to issue a short follow-up post about this. What about the vampires? This post is going in the Data Lab, because it’s for the real sickos out there.

Do not invite the vampires into your poll

The Wisconsin voter file is notorious for having a lot of missing and incorrect data. One of the variables that is particularly errant is the age variable. In their methodology report, for some added context, State Navigate wrote:

“many voters in Wisconsin have a birth year of 1800, and we don’t believe in vampires. These generally are newer voters who skew very young; about half of voters who we polled with a birth year of 1800 say they are in the 18-29 age bracket. Thus, if you were to simply take the voter file’s birth year data, you’ll get a Democratic primary electorate that is majority 65+ and has less than 2% of voters under the age of 30. Absentee requests for the Wisconsin primary (which includes requests for Republican ballots) are 5-6% 18-34. Given that this age group has been notorious for voting late/on Election Day, we find it hard to imagine a Wisconsin Democratic primary electorate that is <8% 18-29. Maybe it is; we don’t know for sure due to the ridiculousness that is the Wisconsin voter file”

Well, I am more than a little wary of using polling data that had too many responses from young people to make any sort of predictions about what age numbers would be on the voter file, so it is another opportunity for sensitivity analysis.

In the plot below, I re-raked the August survey with the under-30 target set by increasingly aggressive assumptions about how old the “vampires” are — from my baseline a priori estimate (unknown ages allocated in proportion to the known ages) up through State Navigate’s estimate (half of unknowns under 30) and past it (70%) — under both the L2C weights and the ideology-free L2B weights.

Notably, stepping up the share of voters who were <30 increases error linearly. Under the L2C weights (which account for ideology and income), the margin goes from Hong +2.4 at my baseline, to Hong +4.9 using State Navigate’s stated floor of an 8%-under-30 electorate, to Hong +6.6 if you assume half of unknowns are under-30, and Hong +8.4 at 70%.

Under the L2B weights (which don’t have ideology in the targets), the same exercise runs from Hong +13.8 up to Hong +19.7, nearly all the way back to the unweighted poll’s +23.8. The reason is simple: the young people who actually answered this survey were overwhelmingly Hong voters (likely too pro-Hong, as the non-response analysis suggests), so every additional young voter you add to the electorate model pushes the poll further from the result.

The adjusted weighted estimates I present — even at a 7-10% under-30 percentage — are still much closer to the result than the original weighted or unweighted numbers. Whatever the true age of Wisconsin’s vampires, the topline takeaways from my original analysis stand: The polls had too many progressives in them, which you can mostly but not entirely solve with weighting — especially if you don’t know the breakdown of the electorate by age! — and a large amount of late movement toward Crowley.

Share

Somebody get the garlic

But about the non-response problem: With the adjusted numbers and new age breaks, the weighted estimates do land farther away from the targets. Does that mean we should conclude there was more non-response among moderates after all?

Well, not entirely. Again, the big problem here is that the Wisconsin voter file can’t produce a good benchmark for age (or other) variables, so pollsters are left flying blind as to what to weight their data to. (The right move here, in my opinion, is to simulate what your poll would show under a bunch of different weights to varying targets, but I will save this for another post.)

To set the stage, here is what the results of the State Navigate poll in August would have been if you allocated undecideds and some minor-candidate voters to Hong and Crowley based on their flows in the July-August recontact study State Navigate conducted. This uses my adjusted weighting model and the maybe-realistic-but-we-don’t-really-know assumption that 50% of people on the Wisconsin voter file with a birth date in the year 1800 are actually under 30 years old (what the polls that had too many young progressives in them indicated).

What is really interesting here is that the base scenario is almost perfect on Hong’s vote share, but underestimates Crowley. That’s likely because there was an additional large swing in his direction after the final August poll wrapped fielding, which polls simply cannot capture. The observed flow to Crowley among supporters of the minor candidates in the recontact study was 1.5:1. I also calculated what the results of the poll would have been if you allocate those voters 2:1 to Crowley vs Hong:

Now, here is the same chart, but if you weight the sample additionally to a benchmark for ideological makeup of the Wisconsin primary electorate. (Remember, the State Navigate poll had a quarter of respondents identifying as Democratic Socialists, which we can be reasonably sure is too high.)

Weighting by ideology recovers about 8 points of vote margin for Crowley.

This means moderates aren’t responding as much to our surveys, right? Well, yes and no. We don’t know if “progressive” is the psychological quality that’s acting on voters to not make them vote. It’s likely something related to one’s ideology: like political engagement. I don’t have the year-vintage breakdown of vote history participation for Wisconsin voters in, say, 2018 (the last time there was a competitive primary in Wisconsin), so I can’t do that math, but maybe someone else could.

But we can use the updated “vampire-adjusted” weights and a reasonable guess at the vote flows in the last week (2:1 to Crowley) and plug into our non-response program (the MUBP — see the original post for an explanation).

The best guess from the authors of the paper I’m basing all this math on is that a reasonable starting point for the impact of non-response is setting phi = 0.5. And you can see why: in the L2Cv weights (which account for ideology), pushing that non-response adjustment up to 100% would have yielded a poll with 20% Hong and 60% Crowley! That’s very bad in the opposite direction of most published polls in Wisconsin.

And to be clear, since the L2Cv weights (solid line) already balance the sample by ideology, the non-response here is something about voting for Hong and Crowley that isn’t proxied by ideological identity in a survey. So you can’t use those trends to say anything about non-response among progressives. That is already taken into account by the level-shift in Hong and Crowley vote at phi = 0.

If, however, you use the L2Bv weights with the non-response dial at the 50% position, you get somewhere around 42% Hong and 40% Crowley. Given they each won about 40% of the vote, that’s a pretty good guess. The residuals here point to about a 5-6 point boost for Hong from non-response.

But again, caution is warranted, since the data for this survey all came from a week before the election. If we had data on election day, the impact of a “non-response” adjustment would be lower. (Technically, I guess, the poll not accounting for future movement in the race is “non-sampling” error, in that it’s error that comes from something other than sampling, but that is different from non-response.)

Share

Polling primaries is very hard

In summary, after adjusting my original analysis for the existence of vampires in Wisconsin, the miss in the original State Navigate polls (+20 points of rHong) breaks down roughly as a result of…

1. 10 points from misspecified age weights

2. At least 5 points of late movement (5 is what we can detect with the recontact

3. About 5 points of non-response

Adjusting for the misspecified age composition gets you within the 13 percentage point average error for the margin between two candidates in primary polls. That would indicate the primary polls and Wisconsin were not any worse than average, after adjusting for what looks like maybe a simple mistake or extra extrapolation from Michigan on the part of State Navigate.

But you can eliminate most of the error in 1 and 3 by weighting your primary poll to a reasonable benchmark for ideological composition. We don’t know what the precise ideological composition of any group of voters is, so see Recommendation 3 in my original analysis. Time for some sensitivity analysis, baby!!

If I were to editorialize a bit, I would say 5 points of ideological non-response in a primary poll is really not bad, all things considered. Especially if it doesn’t impact general election polls or what we know about public opinion writ large.

This post is part of The Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.

Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.

Upgrade to support Strength In Numbers

Following on from last week’s post about the importance of weighting surveys by party identity, a reader asked me whether our new MRP estimates are similarly sensitive to survey weights.

The short answer is no, but this is largely a technicality.

The graph below shows my model’s time-series national estimates of Trump’s approval rating and the U.S. House generic ballot among registered voters since May 2025. The blue line is the published estimates that include Verasight’s survey weights in the regression stage of the MRP model. The red line omits these weights.

These differences are not significant to random variance in the MRP model itself.

So if weighting by party ID matters for surveys, but survey weights don’t impact MRP, what’s going on?

Well, this is largely just a trick I’m playing on you. The MRP model does not weight by party, because it includes a benchmark for the partisan composition of every type of voter in the country, based on their demographics and where they live. The model conditions on those shares to post-stratify by party… which is basically just another way of weighting survey data.

The MRP model I have designed to turn our Strength In Numbers/Verasight polls into state- and district-level estimates of public opinion, explained in detail here, works by first estimating the number of Trump voters, Harris voters, and non-voters as well as the number of Republicans, Democrats, and Independents among every demographic and geographic group in the country. I list these groups separately because they are estimated separately. We have precinct-level results on the former that can be aggregated up to the county level, but we have to estimate the latter.

I estimate that by pushing the raw data for Pew’s NPORS surveys into its own MRP model, which conditions on demographics as well as 2024 past vote, to predict the share of Democrats, Republicans, and independents in each demographic x past vote “cell.” Again, more details in the link above.

The point is that, once we reach the point of running the MRP model on the SIN/Verasight polls, we have already conditioned our projection on party and past vote. We just use “modeling” instead of “weighting.”

But this prompts a few additional questions. First, what if the party ID target used in our post-stratification frame is off? Great question. I take this into account in the post-stratification frame by randomly varying the party ID composition of the electorate in each county by a few percentage points.

Second, what if the Verasight survey were weighted by something not in the MRP model? What if we wanted to estimate not p(vote dem | demographics, geography, party, past vote) but p(vote dem | … + social trust + something else)?

Well, if that something else had no impact on p(vote dem) beyond what could be explained by party and past vote, then we’re in the clear. But figuring that out before an election is, of course, the billion-dollar question. After the 2020 election, YouGov found evidence of political non-response within the party and past vote groups of their panel. This is why communicating uncertainty is really valuable.

In the meantime, the narrow answer to the question we started with is “no, weighting by party doesn’t change the Strength In Numbers MRP estimates.” But that’s because party is in the post-stratification frame.

Thanks for asking!

Share

Subscribe now

My agreement with Verasight (a smart, mixed-mode pollster that does all my polling) is that they get to make all the methodological decisions for our poll while Strength In Numbers drafts question wording and response options. This firewall between editorial and the data-generating process behind the poll is intended to shore up public trust in my reporting; I literally can’t tilt the poll methodology to give me numbers that are more convenient to publish.

But all weighting choices generate a certain amount of uncertainty for polls, and I want to be transparent with readers about how the conventional approach to weighting by party identification could be shaping polling results. This affects poll results for every firm that weights by party ID.

The big question is this: Who counts as a Democrat, Republican, and independent?

There are several ways to answer this question, and each has an impact on a pollster’s results. As we’ll see, Verasight’s data actually fares very well on sensitivity to weighting choices — much better, by comparison, to some other non-probability data collected before the 2024 election.

This post is part of The Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.

Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.

Upgrade to support Strength In Numbers

What is the “right” way to measure party ID?

Pew presents its party ID shares from NPORS in the following graph:

You could count up Democrats, Republicans, and independents in three ways using this data:

1. Count independents who say they “lean” toward one party as belonging to that party, and sum up party shares (47D, 43R, 10I). This is what Pew does.

2. Count independents who say they “lean” toward one party as independents (28R, 29D, 44I).

3. Treat “leaners” as their own group (29D, 19 Lean D, 10 I, 15 Lean R, 28R)

Currently, Verasight weights the Strength In Numbers data using the second methodology listed above, weighting to what I’ll call from now on “3-way party ID” instead of “leaned party ID”. There’s also “5-way party ID” (bullet three above).

So how would our results change if we weighted to leaned or 5-way party ID instead of 3-way? And how do our results change in 2026 if we update the weights from their 2025 to 2026 vintages? (That’s the question I had originally that set off this whole investigation.)

To answer these questions, I reverse-engineered Verasight’s weights for our published Strength In Numbers polling using its weighted demographic and partisan marginals. Then I updated those weights in three separate experimental polls to account for the three new party ID benchmarks. Fake poll A re-weights the published data to 3-way party ID per the 2026 NPORS alone — the same counting rule we use today, just the new vintage. Fake poll B re-weights to leaned party ID per the 2026 NPORS. And fake poll C re-weights to 5-way party ID in the 2026 NPORS. So the gap between the published line and fake poll A is the vintage decision, and the gaps among A, B, and C are the leaner decision.

To account for uncertainty in Pew’s party ID benchmark (their NPORS survey reports a margin of error of 1.8 percentage points in 2026), I repeated each experiment 500 times while randomly varying the party ID results used in the weighting program.

Share

Our polling data, weighted four different ways

The chart below shows the generic congressional ballot among registered voters under each of the three experimental weighting models described above, compared to the average published result from May through July 2026.

Here’s how I interpret this graph:

• Baseline published result is D+7.1

• NPORS shows an aggregate party ID margin of R+0.7 across 2023, 2024, and 2025 — the reference for our published result. Updating the 3-way party ID target alone to D+1 (fake poll A) lifts the generic ballot to D+7.9. The updated party ID targets are a no-brainer, so that’s a free +0.8 points for Democrats.

• Taking independents who lean toward one party and putting them into their partisan group and weighting to that benchmark (47D, 43R, 10I) lifts Democrats’ margin further to +8.7.

• If you balance your sample by 5-way party ID as well (pinning the leaner percent to Pew), the result rises again to D+9.2.

All told, the gap between our published number and the most aggressive party pin is about two points of margin.

For some added context, Trump’s July approval moves from 38%–59% to 37%–61% under the five-way pin, a net swing of two points against him.

To understand why the poll that’s weighted to 5-way party ID is so bullish on Democrats, it helps to understand how Pew’s NPORS party ID breakdown currently differs from the data Verasight sends Strength In Numbers. Under our current weighting approach, the July poll’s three-way split already matches NPORS to the decimal point — with about 30% Democrats, 31% Republicans, and 40% everything else.

But the political makeup of those independents varies a lot between the samples. According to Verasight’s July weighted estimates, 13% of U.S, adults are independents who lean toward Democrats, 10% are Republican leaners, and16% pure independents. Pew, by contrast, has far more leaners on both sides — 19% and 15% — and far fewer pure independents, at just 10%. Here’s a visual comparison:

Weighting to the five-way benchmark upweights both leaner groups by roughly half and cuts the pure-independent share by a third. It also trims the party totals — about three points off the Republican share — because the 2026-only vintage comes along for the ride: a three-year average still carries 2024 and 2025, when Republicans were a bigger slice of adults. But the biggest change is the composition of the middle.

One question you might have, however, is why updating the party ID pin is only worth about ~2points on margin for Democrats, when Pew’s NPORS data shows the nation moving about 4 points left over the last year.

The reason is that the poll is also weighted by recalled 2024 vote, which provides another source of balancing by partisan attributes. So when the new party target pushes weight out of the Republican column and into the independent one, the weighting still has to produce a sample that voted Trump+1 in 2024. It makes up the difference by upweighting the Trump voters inside the newly enlarged independent bloc. The party pin gives; the 2024-vote pin takes most of it back.

We can prove it by deleting the 2024-vote pin and re-running fake poll A. Without that anchor, updating the party target to 2026 moves the pooled margin from D+7.1 to D+10.0 — a shift of nearly three points, right in line with the naive arithmetic. But it gets there by drifting the sample’s recalled 2024 vote from about Trump+1 to Harris+2, a 2024 electorate that technically never existed. That’s exactly the failure the vote pin exists to prevent. After the weighting model weights by both party ID and past vote, it’s left drifting up “just” to D+9.

However, the statistical cost is of all this weighting is real, if modest: the design effect of the weights (a measure of added variance from the underlying balancing model) rises from about 1.04 to about 1.15 in a typical wave, which trims the effective sample size of a 1,500-person poll by roughly 125 respondents — or, a widening of uncertainty intervals by about five percent.

And there’s the additional uncertainty from balancing by multiple groups simultaneously, as in the 5-way party ID pin. NPORS is a survey with its own margin of error; a proper margin of error for our survey should take into account that Pew’s estimate of 19% for “lean Democrat” really means somewhere between about 17% and 21% — and you can repeat that exercise for every slice of the target we’re weighting to.

As mentioned above, I account for this by randomly varying the party ID target I weight our Verasight data to by the amounts implied by Pew’s sample sizes for each party ID group. Each of 500 alternative NPORS distributions is fed into a computer program that re-weights the pooled poll to give a final margin of error that combines uncertainty from both surveys. This contributes a further 0.9 points of standard error to the five-way margin, about 0.7 to the leaned margin, and roughly 0.5 to either 3-way margin (which pin fewer categories with larger sample sizes). Adding the two in quadrature and doubling for a 95% interval gives a total margin of error of about ±2.9 points on the published and 3-way 2026 margins, ±3.2 on the leaned margin, and ±3.4 on the five-way margin.

The upshot is the more precise you are trying to be with your party ID weighting scheme, the more potential statistical error you are adding into your poll.

Does weighting by party ID make a poll more accurate? (Yes.)

Unfortunately without a crystal ball, we cannot know which weighting scheme will make our poll more accurate for 2026. But we can see which approach would have been most accurate in past elections.

To figure this out, I took the raw data from the 60,000-person Cooperative Election Study surveys from 2024 and 2020 and reweighted them the same way I described above: Each year’s poll was weighted to be representative by Census demographics, then given a weight according to its party ID in three ways — party pinned 3-way, leaned 3-way, and 5-way, using the NPORS distribution from each cycle.

The results are shown in the graphic below.

In deciphering the above, start with the bottom panel. Trump won 50.7% of the two-party vote, but when weighted to demographics alone, the CES gave him just 43.2%. That’s a whopping 7-point bias in vote share, which would make for one of the worst national pre-election poll misses since 1952.

But weight by party, the picture changes dramatically.

The 3-way party weight recovered most of the gap between the result and original estimate, putting Trump at 48.4% of the two-party vote. The leaned and five-way pins got within a point — 49.9% and 50.0% — and on post-election recalled vote (not shown) both were essentially exact.

Weighting by party was less efficacious in 2020, and unlike in 2024, weighting by leaned party was not additionally helpful in reducing bias; nothing fully fixed that year’s Democratic lean. Weighting by demographics alone put Trump at 41.9% against an actual 47.7%. The three-way party weight got him up to 45.2%, and the leaner pins added just two tenths of a point.

The conclusion from the historical data is that weighting by party is a must, but weighting by leaned party is not a proven solution to reduce bias. The only thing we know for a fact the extra weighting does is increase sample design effects and thus uncertainty. It makes sense that some pollsters would not want to make that tradeoff (especially if they were designing their methodologies before 2024!).

For good measure, I ran one final test on our own Strength In Numbers/Verasight July sample: I held out the 2024-vote benchmark from the weighting model and calibrated the survey by party in the three methods above. Those results are shown below:

When weighted just to demographics, Verasight’s sample is off (though meaningfully less biased than the CES) when predicting the results of the 2024 election. The conclusion from this is that weighting by some political variable is necessary. Weighting by unleaned 3-way party ID does not reduce much bias, suggesting that the independents in the sample lean Democratic by past vote. The five-way party break with pre-election 2024 NPORS numbers recovered the result of the election almost perfectly, while the leaned three-way was off by less than a point.

Still, note that all weighting specifications came within the margin of error of 2024 recalled vote, and of each other.

Share

Weighting by party is not black and white

So why not switch tomorrow? A few considerations.

First, pinning the leaners only makes the poll more accurate if the leaner benchmark is right. Unfortunately, while NPORS is a high-response rate survey developed by some of the smartest minds in polling, it is still only a survey. That means it’s subject to the same sampling and non-sampling error every other poll is. If the leaned party ID is off by 2 points, all polls that weight to it will also be off. Verasight uses a three-year running average of three-way party ID to minimize potential error in the benchmark, which I think is a sensible cautious approach.

And second, NPORS only comes out once a year. Even a perfectly calibrated current target gets stale over time. The 2026 NPORS was fielded from January to June 2026. By Election Day, the entire sample will be 5-months out of date, and the oldest interviews will be nearly a year old! That’s a lot of stale data exactly when polling accuracy matters most. The three-way mix is less likely to go wildly out of whack than the 5-way.

This analysis shows that weighting by some benchmark for party is a good idea. We have done that. The second-order questions (3-way leaned or 5-way split? what about 7-way?) are less important and unresolved.

Finally, there’s the editorial factor I started with: Verasight develops the methodology for the Strength In Numbers poll, not me. If they decide it’s best to update their approach, I’ll follow their lead.

Subscribe now

Got a weighting or methods question you want the Data Lab to dig into? Leave it in the comments below!

Related Articles

This is the inaugural post for the Strength In Numbers Data Lab, a new section of this publication for sharing extra charts, deep polling nerdery, and writing about methods work etc. for the devoted reader of this newsletter. If you download the crosstabs of our monthly Strength In Numbers/Verasight poll, this one’s for you.

The idea for this section is simply to publish more, less polished, and more technical content, both because I know there are many people who are hungry for more content and because there’s stuff I want to keep off the full email list. See, a lot of the analysis I do behind this scenes never makes it into a proper article suited for 70,000+ readers, whether that’s because of the editorial calendar, it’s just one chart, or a reader asks a question that deserves a real answer but is too niche for most subscribers. I tend to file these extras away or send them off as emails where they die in private. That’s a big no-no from a blogging point of view (Always Be Posting) so from now on I’m going to be sharing that work here, outside the Strength In Numbers home vertical (and your inbox) but still public for the real sickos.

Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here.

Like the main site, most Data Lab posts will be free, but I’ll reserve the occasional longer article for paying subscribers as a bonus for making all of this possible. Upgrade your subscription for full access.

Upgrade to support Strength In Numbers

Finally, as the staff here at Strength In Numbers continues to grow, I’m hoping the Data Lab will also be home to contributed musings from our survey research assistants and other polling professionals mining SIN data for insights. If that’s the type of thing that interests you, email me.

Short programming note: I’m coming back from paternity leave in two weeks, on July 27, when Strength In Numbers will return to full editorial strength and I’ll have the results of our new July poll to share. David and I return for our first summer podcast/Substack live show that Thursday, July 30. Get signed up so you don’t miss out!

With all that out of the way, let’s dive in:

What’s up with people who disapprove of Trump but don’t vote Democratic?

After my May piece on cross-pressured voters, a reader named Laurence asked on the SIN Discord for me to run some special numbers on our polling microdata. That piece was written in response to some bad punditry about Democrats’ midterm polling — the broader point I was making was that you shouldn’t expect everyone who disapproves of Trump to vote for Democrats for Congress, and grading them against that expectation was unfair/misleading

Laurance wanted me to take the Trump disapprovers who still aren’t voting Democratic and carry at least one Republican “anchor” — a 2024 Trump ballot, Republican party ID or lean, or conservative self-ID — and split them up by which anchors they carry. How does each combination vote on the generic ballot? In other words, don’t just count the anchors — ask which ones actually hold.

I recommend you read the piece linked above to get a better idea of how this works, but basically what we’re doing is looking at the subset of people in our poll who say they disapprove of Trump but aren’t voting Democrat and trying to figure out why — and, importantly, how many of those voters Democrats might actually win.

To answer Laurence’s question, I stacked all six of our Strength In Numbers/Verasight monthly polls from January through June 2026 into one big mega poll. This gives us 8,818 registered voters and enough sample to slice the disapprovers thinly.

And here’s where the microdata is really powerful (and why I think what we’re doing here at SIN is so cool). If you sort voters by Trump approval and generic-ballot vote and then count the GOP anchors in each cell, we find that among disapprovers who are undecided, 40% carry one or more GOP anchors. Among disapprovers who say they’ll vote GOP, 90% have Republican anchors. That’s the group I called “closet Republicans” in my May piece. There are a lot of disapproves who look more like Trump supporters/Republicans than Democratic voters, all else being equal.

Now to Laurence’s actual question: here’s the generic ballot for each anchor combination, split by Trump approval:

This chart is great, even if it takes a while to grasp, because now we can see how your anchors condition your behavioral response to Trump’s presidency. Disapproving of Trump when you’re otherwise a solid Trump voter doesn’t mean you’ll become a Democratic voter in the midterms.

Here are the numbers: Voters carrying all three anchors (a Trump ‘24 ballot, Republican in party affiliation, and conservative self-ID) are still voting Republican 76% to 16, so disapproving of Trump barely dents them. Voters who pulled the lever for Trump in ‘24 and call themselves Republicans but not conservatives stay Republican too, at 63% to 26.

But drop the party label and everything changes. Disapprovers who voted for Trump in 2024 yet call themselves neither Republican nor conservative now go Democratic 62% to 22, an outright flip. The Trump-plus-conservative-but-not-Republican cell (second column, second row) points the same direction, at 50% Democratic to 29 Republican (though it contains just 24 respondents across six months of polling, so treat that finding as directional).

The pattern is hard to miss. The subgroups that keep “Republican” in their identity stay Republican by 37 points or more in the generic ballot even while disapproving of the man leading their party, — while both subgroups without it defect to the Democrats by a margin of 30-plus points. Party identity is doing a lot of work, even among Trump voters.

This is why I called this group “closet Republicans,” and I think that characterization stands.

Share

The usual caveats apply: this analysis pools six monthly waves of data, so some of the underlying data may be out of date, and some of these breakdowns have only a few dozen voters in them so I wouldn’t take the exact estimates to the bank. But I think the broader point about GOP identity stands — and, more to the point of the Data Lab, I think I’ve answered Laurence’s question now!

If you have a question you want me to answer with our poll, leave it in the comments below or on the SIN Discord and it might become a future dispatch.

Subscribe now

Related Articles

Welcome to the Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit neatly into the normal editorial vertical of the full site/doesn’t get the full editing treatment before publication. If you’re familiar with laboratory science, the idea is basically to open-source the lab notes for all the polling I run and explore stuff like survey experiments in more detail. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.

The Data Lab will also be a place for general miscellany that committed readers will be interested in, but might marginalize the wider audience. I also cut a lot of charts from the full newsletter that I’d like to share somehow, and have some insights into running a Bayesian model-driven polling operation and independent media company that I think folks will find interesting. Finally, as the staff here at Strength In Numbers continues to grow, I’m hoping the Data Lab will also be home to contributed musings from a network of survey research assistants and other polling professionals mining SIN data for insights. If that’s the type of thing that interests you, email me.

In terms of content mix, the goal is that well above half of Data Lab posts will be free. Occasional deeper dives will be paywalled for members, whose support makes all of this possible. Upgrade your subscription for full access:

Upgrade to support Strength In Numbers

The Data Lab is intentionally opt-in for the true Strength In Numbers sickos, so if this at all sounds interesting to you, go to your settings for this publication and opt in to emails from the Data Lab. You should see a menu like the one in the screenshot below with a button to click to enroll in the new vertical. (And you might as well sign up for podcast notifications while you’re at it!)

The first Data Lab post is coming this week. If you have a suggestions for something else I should write, or a question about my polling, leave that in the comments below! Comments are unlocked for all readers.

Leave a comment

We (Elliott + Verasight, my survey partner!) just went into the field with our January poll, and part of the process for that survey was compiling all the questions y’all sent in so we could pick a few to run. For transparency (and fun!), I figured I’d share the complete list of your questions below. I have done a very rough edit of the question text and response options for clarity. Not all of these questions will be used this year, and most are not in the final form we would use in a poll; I’m just sharing them for fun.

Happy reading. If these spark any additional questions or comments, drop ‘em below!

Share

Leave a comment

Read more

I have activated a special coupon for the 2025 holiday season. Click the button below and get 20% off an annual subscription to Strength In Numbers, no questions asked. Perfect for new subscribers, monthly subs looking to save some money annually, or as a gift to a loved one!

Get 20% off for 1 year

Today I have a short, interactive post for the whole audience.

One of the things that sets this Substack apart from other political analysis blogs is that Strength In Numbers sponsors its own polling, conducted in partnership with a mixed-mode polling firm called Verasight (“mixed mode” means they source interviews via multiple modes: in this case, via the internet, text messages, and phone calls). And we (Elliott + Verasight) also let subscribers to the site suggest their own questions for us to ask 1,500-2,000 U.S adults each month.

As far as I know, SIN was the first major politics publication on Substack to make bespoke, audience-driven polling a regular part of editorial strategy. This turned out to be one of the best ideas I had in 2025; the polls generate a lot of back-and-forth with readers (I try to respond to every one of you!) and drive press coverage for the website. The idea was so good that others have followed in our footsteps! Yay for independent polling!

These monthly surveys generate a lot of insights beyond the normal toplines and crosstabs you see from many polls. Because I am a pollster and a statistician/data scientist/computational social scientist at heart, I’ve been able to use the raw data Verasight gives me to do a lot of cool things, including visual analysis of open-ended text responses and estimation of public opinion at the state level (and, soon, maybe even counties and congressional districts). Of course, there’s also everyone’s favorite feature of the monthly poll: line charts.

So far, I have devoted about half of each survey to a core set of tracking questions I ask again and again each month. Then, the rest of the survey is filled in with rotating trackers when they become relevant (tariffs, for example, have been in and out of the news since January), plus one-off questions submitted by subscribers.

And this is where you come in.

What questions do you want us to ask in 2026?

I want everyone who reads this newsletter to suggest one question you’d like us to track on a regular monthly basis in 2026. These questions should be suited to trending (ie, visualizing on line charts) and relevant over the course of the whole election cycle (January - November; we’re going to take December 2026 off). The comments are open below for your suggestions.

The questions I’ve been regularly asking so far are:

• Self-reported likelihood to vote in midterms

• U.S. House generic congressional ballot

• Most Important Problem for 11 issues:

  • Jobs and the economy

  • Prices/inflation

  • Trade with other countries

  • Foreign policy

  • Immigration

  • Border security

  • Deportations

  • Education

  • Health care

  • Government funding and social programs

  • Crime and public safety

• Party and political figure feeling thermometers

• “Which party do you trust most to handle ___” for each of the aboved 12 issues

• Trump approval

• Trump approval for those 12 issues

• Right direction/wrong track

And I plan to add the following questions to the regular rotation:

• Do you think your personal financial situation will get better or worse in the next year? (Better / worse / stay the same / don’t know)

• Two open-ended questions asking what single issue Trump is doing best/worst on. (Used to see if I should add or subtract an issue from the issue tracking.)

• What news story did you hear the most about over the last month?

My goal next year is to somewhat dramatically expand the data products produced by the data in the monthly poll, including making regularly updated custom graphics from the interviews we conduct. That would include crosstabs, automated graphs for trended questions, analysis of open-ended text, and MRP modeling for issue questions. We are also planning on sharing the data with academic data archives, and maybe even open-sourcing some of the analysis pipelines on GitHub (stay tuned).

Full disclosure: Some of these features will probably get locked behind a paywall at some point to help grow the business, but I intend to keep the vast majority of insights from the polls freely accessible to the public.

So, please fire away! I want to know what questions you are curious about, so I can provide an interesting and actionable survey to readers. The whole reason we started this project was to make polling more transparent and relevant to the average American.

When you’re writing your question, keep these guidelines in mind:

• Please only submit one question each. I promise to read every submission, but that’s going to be a lot of work already (SIN is closing in on 50,000 readers), so please don’t abuse the comments.

• Make it one concept per question. Avoid “and/or” mashups; they’re harder to interpret and trend cleanly

• Try to use neutral wording. It is important that questions do not lead respondents to answer a certain way. We’ll lightly edit for clarity/neutrality and keep wording stable over time for trend comparability.

• Include response options if you can. Best formats for monthly tracking are strongly/somewhat approve/disapprove, better/worse/neutral, right/wrong track, favorability scales, 4-point “a lot/some/not much/none”, and scales of numeric integers.

Leave your questions below — or, if you’re a member of the Strength In Numbers Discord, tag me there.

Elliott

Leave a comment

Share