This post is part of The Data Lab, a new section of Strength In Numbers for extra charts, general polling wonkery, and methods work that doesn’t fit into the normal editorial vertical. If you’re the type of person who wants to know how our polling is done or reads election forecasting methodologies, the Data Lab is for you.
Emails for posts on the Data Lab vertical are off by default, but you can easily toggle them on here. If you have a post to contribute to the Data Lab, email me.
Most Data Lab posts are free, but the occasional deeper ones are for paying subscribers, whose support makes all of this possible.
The folks behind the Marquette University Law School poll have responded to my analysis of last week’s polling miss in the Democratic primary for governor of Wisconsin. As a reminder, the final Marquette poll of the race from July 22-27 had Francesca Hong winning 38% of the vote and David Crowley, who ultimately won with 40% of the vote, at 7%. This is a 33-point miss — a “remarkable difference between poll and outcome,” per the director of the poll, Charles Franklin. In margin terms, this is a 31-point miss.
As a second reminder, my analysis looked at the microdata from a non-profit I did some poll methods work for previously called State Navigate, whose August poll had Hong at 44% and Crowley at 20% — a 20-point miss, about 50% less than Marquette.
The explanation from Marquette of their miss is here. Part of their explanation is also that their poll was taken early, before a lot of late consolidation to Crowley. That’s plausible, but also, 31 points is a lot of consolidation. This is one of the hard things about conducting primary polls in general, though; preferences are very fluid and tend to move a lot very late, especially when candidates are entering and exiting the race.
I don’t want to get into a line-by-line back and forth, but I do want to put their opening here. Marquette’s piece begins as such:
G. Elliot Morris argues that the polls in Wisconsin underestimated David Crowley’s support in the 2026 Democratic primary because they included too many young voters (who mostly supported Francesca Hong) and too few older voters (who mostly supported Crowley).
I think a more accurate reading of my article is that the State Navigate survey was primarily biased because they included too many young voters (with a target for the under-30 population ranging from 9% in their August poll to 17% in July. Additionally, in my article I gave three reasons why the polls were off, not just one:
Most of the miss in polls in Wisconsin is attributable to faulty demographic targets (too many young people). This inflated Hong’s vote margin by somewhere between 5 and 10 points.
My best guess is that the race moved 6-10 points toward Crowley after pollsters released their final surveys.
Non-ignorable non-response within demographic categories likely further inflated Hong’s vote margin by 2-5 points.
Fair enough, I did say “in polls” there, though the article was mostly about State Navigate. I think this probably made Marquette think I was attacking them and hurt their feelings, though, which was not my intention. But note I did attribute the lion’s share of the “error” to late movement, as does Marquette’s John Johnson:
I would not describe late movement as leading to “error” in our survey, since we were measuring the electorate as of July 22-27 and not predicting what it would be on August 11.
The remainder of Johnson’s article is a good deep dive into why it’s very hard to know what the age composition of the likely total electorate in Wisconsin is. That’s because the state of Wisconsin does not make age of registrant available for commercial voter file vendors, so you get the “vampire” problem State Navigate termed and which I described in the footnotes of my original article, and in this follow-up.
I’m pretty sure Johnson didn’t read the footnotes or follow-up because I did a sensitivity analysis of the age makeup of the Wisconsin electorate that answers his objections outright. I explained that we didn’t know from the WI file what the unknown % exactly was, that polling data provided a benchmark of about 50% - 75% of the entrants with unknown ages being in the 18-30 camp, and then tested what the analysis would show assuming those percentages. I demonstrated that the poll bias was not linearly related to the age composition; that is, even at a high 18-29 voter share, the weighted State Navigate polling data was still a lot better than what they published in August. I’m sure it would make a big difference vs what they published in July,
I explained, for example, that even if you weight the State Navigate data to a 9% target for ages 18-30, you still get a big Hong overestimation (this data is from August 7-8, the week before the election):
Importantly, I also showed that weighting to a rough guess of the ideological composition of Wisconsin Democratic voters, via 2016 primary exit polls, gave a much more accurate picture of the race (that’s the solid line above).
But none of this answers the big question: What share of the Wisconsin likely primary voter universe is made up of under-30s? We will never know for sure, so we are left guessing — and, importantly, weighting polls to those guesses, which is why sensitivity tests of the kind above are so useful and why pollsters should publish them.
For some informed guesswork, Johnson produces this table of estimates of all registered voters and general election voters using data from the Current Population Survey. It shows that 14% of self-described 2024 general election voters in Wisconsin were ages 18-29.
This appears to be slightly less than the benchmark Marquette weighted to (18% in their published tabs). I think this is because their poll weighted to registered voter benchmarks, and then filtered to the population of likely voters based on their stated intent. (This introduces a lot of noise because people are really bad at estimating how likely they are to vote.) I have asked Johnson on Substack to give me the exact weighting targets and will update this paragraph when he responds.
Unfortunately, I’m pretty sure both of these estimates from Johnson are too high. For one thing, a poll weighted to a lower share is more accurate in predicting the winner, even if you adjust for ideology and for late deciders favoring Crowley. (But maybe that’s just because those weights deflate Hong supporters, but still give an unrepresentative look at the electorate. Fair enough.)
The problem with Johnson’s estimates is that he is looking at a different population than I was. For re-weighting the State Navigate data, I was targeting the population of Wisconsin voters who have cast at least one ballot in a Democratic primary for any office since 2016. Johnson is, in contrast, producing an estimate of the statewide population of likely November presidential general election voters. The latter population is going to have a lot more young people in it, because primary voters skew old.
To demonstrate that point, I have loaded up the L2 voter file for Minnesota, which does not have missing data for voters by age, and is demographically one of the most similar states to Wisconsin. According to L2, the statewide share of registered voters in Minnesota who are between the ages of 18 and 30 is 18%. However, the share of likely Democratic primary voters who are between 18 and 30 is only 11%. That’s a 7-point difference. Additionally, about 14% of validated 2024 voters were 18-30 — 3 points higher than the likely primary electorate.
And according to the Census, Minnesota has about 2 points more voters who are 18-30 than Wisconsin does. (Wisconsin voters are older than Minnesota voters.)
If we take Johnson’s estimate that 14% of Wisconsin 2024 voters were ages 18-30 and subtract 3 + 2 from it, we get 9%.
This is the same target I used in the updated analysis I posted last Thursday, which “controls for the vampires.” In that analysis, I produced the following breakdown of error in the State Navigate data, using the 9% 18-30 target:
10 points having too many young people people in the poll / other weirdness in the State Navigate weights
At least 5 points of late movement (5 is what we can detect with the recontact) and plausible up to 10
About 5 points of non-response bias
Adjusting for the misspecified age composition gets you within the 13 percentage point average error for the margin between two candidates in primary polls. That would indicate the primary polls and Wisconsin were not any worse than average, after adjusting for what looks like maybe a simple mistake or extra extrapolation from Michigan on the part of State Navigate.
But you can eliminate most of the error in 1 and 3 by weighting your primary poll to a reasonable benchmark for ideological composition. We don’t know what the precise ideological composition of any group of voters is, so see Recommendation 3 in my original analysis. Time for some sensitivity analysis, baby!!
For a few more examples, Strength In Numbers subscriber and the former Secretary of State of the state of Oregon, Phil, calculated the following turnout and young voter percentages in recent primaries in the following states:
NC - 21% overall turnout, 18-29s 7% share
OR - 42% overall turnout, 18-29s 6% share
PA - 22% overall turnout, 19-29s 8% share
OH - 23% overall turnout, 18-29s 7.5% share
He wrote me:
With 35% overall turnout in WI, my educated guess is that the 18-29 year old share of the D electorate might be closer to 10%, plus or minus 2%.
In summary
I’m sorry if I hurt Marquette’s feelings.
I already conducted a sensitivity analysis on the youth composition of Wisconsin voters, which I should have just published up top in the first article so I didn’t confuse people, and the conclusions about error do not change.
Primary polling is very hard, especially in states where we don’t have good voter file data on age composition. People should adjust their expectations for accuracy downward, especially in polls taken weeks before the actual election (as in the case of Marquette).
Marquette’s alternate estimate of the age composition of Wisconsin seems too high; this is because they are not adjusting for the fact that likely primary voters tend to be older.
If I were you, I would just put this on repeat:
Primary polling is very hard, especially in states where we don’t have good voter file data on age composition. People should adjust their expectations for accuracy downward, especially in polls taken weeks before the actual election.
A great pollster is:
*a political scientist,
*a philosophical economist,
*a numerical psychologist,
*a visual/artistic mathematician, and
*an inexact prophet.
You are great. Thank you.
Thank you for the detailed rebuttal/revised summary-- and for making me laugh out loud about the feelings.