Why so much election analysis is basically useless
A race decided by a point and a half can be blamed on almost anything — and that severely limits what we can learn for the future
Last Friday, I (reluctantly) wrote a long piece about the Democratic National Committee’s finally published “autopsy” of the party’s 2024 campaign. In that post I argued that the single biggest cause of Trump’s win was the economy/inflation — and that it was telling this was mostly not reckoned with in the report (the word “inflation” isn’t mentioned at all in the published version). If you’re not addressing the biggest factor in your party’s loss, you are limiting the lessons you can learn from your look back at its campaign. The report also had a lot of examples of the Strategist’s Fallacy that decrease utility.
Perhaps this is why Ken Martin, the Chair of the DNC, didn’t want to release the report in the first place; the most robust answer to the question “Why did Democrats lose in 2024?” (answer: “inflation”) offers few to no action items going forward.
But another point to make about the 2024 DNC autopsy — and a point I had to drop from the original article (it was already quite long) — is that the closeness of the election makes it impossible to determine what really cost Democrats support. For this week’s Chart of the Week (the last one before I take a short paternity leave — don’t worry, I’ve pre-written six weeks’ worth of deep dives to come out in June and July!) I wanted to write another brief but important article in part about the DNC’s 2024 report. In particular, I think we should zoom out and reconsider the autopsy as an analytical product altogether. The reality is that we can’t really learn much from campaign autopsies in close races — and this leads people to adopt a lot of beliefs about party strategy that don’t hold up in the future. That’s genuinely pretty damaging for parties, so they should rethink their approach to studying the efficacy of campaigns.
This article is free to read, but it was not free to write. Subscribers to Strength In Numbers get exclusive data-driven analysis of politics and elections, access to premium polling data, and support independent, interactive data journalism. Consider joining as a paying member today:
If a paid subscription is not in the cards for you, the best thing you can do is share this article with a friend. Word of mouth is the #1 way new people learn about this publication! Thank you.
The 2024 election was really close, and that limits what we can learn about what worked
Donald Trump won the “tipping-point” state (the one whose electoral votes push the winner past 270, if you order the states by Democratic/Republican vote margin) by just 1.7 percentage points in 2024. Trump defeated Kamala Harris by just 120,000 votes in Pennsylvania. The excess electoral votes he won in Michigan and Wisconsin were decided by 110,000 votes — a fifth of the size of the smallest state.
The problem this presents for election analysts is that every campaign decision becomes potentially decisive for the outcome. There were 152 million votes cast in 2024, but the contest was decided by just 230,000 in those three states above — or 0.15 percent of all voters. When a race is that close, almost anything plausibly mattered. Inflation, as I wrote most about last week, could have flipped it on its own. So could Joe Biden clinging on until July, the administration’s policy on immigration and Israel/Gaza — or even soft turnout in Detroit and a few rainy days in Pennsylvania. Anything could have mattered — which means everything did.
Another way to put this is that the smaller the margin, the more explanations become mathematically plausible. In a 10-point loss, you need a large, systematic failure to explain the result. In a 1.5-point loss, you don’t. Empirically speaking, factors such as presidential popularity, wars, inflation, GDP growth, real disposable income, and candidate favorability all have effect sizes for vote margin that are at least double the size of Trump’s winning margin in 2024.
And that doesn’t even cover those factors we can’t study quantitatively: A slightly different nominee-selection timeline in 2024, a smaller turnout decline among core Democratic voters, a few thousand more persuadable voters breaking left, or a marginally worse Republican ad campaign could each have pushed the election the other way. The problem is not that these explanations are wrong. The problem is that we can’t use the result of the election to order them by influence.
Close elections are what social scientists call overdetermined — that is, the product of too many factors to decipher what dominated. This means they are also underpowered as causal evidence; when everything was enough to cause the loss, nothing is uniquely to blame for it. The election just doesn’t carry enough information to tell you which factor was the factor.
When this happens, people resolve the uncertainty about what matters by writing takes that reaffirm their priors. Several high-profile Democratic Party donor groups and centrist think tanks boosted the autopsy (with its scant new data and research) as additional evidence that their own ideological agenda is the only way forward (despite the new data that says this is overconfident). That’s not analysis so much as projection.
Worse, the results of the election can also be used to argue mutually incompatible conclusions. If you think Democrats are too moderate, you can point to youth and nonwhite erosion and say the party failed to inspire its base. If you think Democrats are too liberal, you can point to losses among working-class voters and say the party alienated the middle. If you think campaigns matter, you can point to the relative Democratic resilience in battleground states as evidence the Harris campaign worked (this is, I think, broadly true). If you think campaigns don’t matter much, you can point to the national anti-incumbent swing as evidence of the relative futility of the campaign as well (paradoxically, that’s also true). You can see how limiting analysis becomes when everything mattered.
The opposite problem also matters: autopsies are bad at noticing what worked. If a party loses, the review is naturally organized around failure. But in a close election, the more revealing question is often why the loss was not worse. That was the question I dug into in part in last week’s post.
There’s also a risk of over-learning the wrong lesson
Our desire for clean, simple stories about the world leads to a related problem with autopsies: a tendency to hold on to single stories that extrapolate poorly or otherwise age badly.
The best example of this is the Republican Party’s autopsy of Mitt Romney’s loss in 2012. The conclusion from that report told Republicans to embrace immigration reform and soften their tone toward Latinos to win over more voters in a diversifying electorate.
The reason this example matters is that the 2012 autopsy was not obviously unserious. It was not a stray consultant memo or a cable-news hot take. The RNC billed it as a sweeping review of what had gone wrong, complete with recommendations on outreach, data, digital, the primary calendar, debates and immigration. Among other things, the report called for a $10 million minority-outreach push, comprehensive immigration reform, fewer primary debates, an earlier convention and a stronger data and digital operation. It would have completely remade the campaign goals and operation of the RNC.
And yet, the 2012 RNC autopsy fell apart almost immediately at the next election cycle. In 2015, Trump rode down his golden escalator and implied immigrants were rapists and criminals, and then pursued the harshest immigration policy in a generation — the opposite of everything the report advised. He won anyway. And then, from 2020 to 2024, the GOP’s share of the Latino vote climbed from 36% to 48%, per Pew.
That doesn’t mean harsh immigration rhetoric was the secret to winning in 2016 and 2024. It almost certainly wasn’t. The point is narrower and more important: strategic narratives built from a single election often age badly. The RNC autopsy treated its most notable failing as the central lesson of 2012, when there were many other reasons Romney struggled. Four years later, the party won the White House with a very harsh message on immigration. Eight years after that, its nominee made large gains among the very voters the autopsy had treated as requiring a softer message to convert.
And, not for nothing, after 2020 there was no (public) GOP autopsy at all. (I guess you can’t analyze a defeat your party leader denies happened at all.)
It’s fair for people to want answers from the parties about what they could have done better. These organizations spend (or organize the spending of) billions of dollars every four years; what are they doing right or wrong with that money? People want certainty about what to do next time.
Unfortunately, autopsies simply can’t provide that certainty. A close loss like the Democrats’ in 2024 doesn’t come with one mono-cause; it comes with a dozen, or even dozens. You might as well just randomly pick one of them to fix for the next campaign than spend even more time and money proving the “most important” ones.
And 2024 wasn’t a one-off. Close elections are now the rule, not the exception. The tipping-point state has been decided by three points or fewer in five of the last seven elections.
Based on the other national elections this century, plus the increasing partisan sorting of the electorate, odds are that the 2026 and 2028 elections will be close, too.
None of this means campaigns shouldn’t review their decisions. They should. They should test ads, audit field programs, evaluate polling methodologies, compare persuasion and mobilization experiments, and figure out which operational failures were preventable.
But this isn’t really what most people are doing when they’re looking back at campaigns. They are trying to sell you a magic solution — their magic solution — to the problems that most animate them. And in all fairness, in a close election, the data gives lazy analysts plenty of ammo to work with.
But if you are to read such reports, you should understand them for what they are: noise. Especially in close elections, party autopsies are mostly a distracting exercise in futility.
If you read this far, I’m confident you would get a lot out of a paid subscription to Strength In Numbers. You’ll get weekly premium posts like this one, access to the full historical archive of posts here on Substack, and early access to new data products — plus the ability to add questions to our monthly surveys. Read more here or subscribe using the button below:
Great piece. The takeaway for me is that Dems can’t afford to indulge their innate tendency to focus on idealistic campaigns focused on policy papers and the antiquated, ridiculous notion that voters will credit them for past accomplishments. Voters are largely uninformed, self-interested and extremely vulnerable to the barrage of propaganda and misinformation we are subjected to 24/7. Dems must be smarter about social media and messaging if they want to have any chance of implementing their idealistic agenda.
A great piece. Should be required reading for every podcast bro.