Poll Back the Curtain
Why Polls Aren't Always What They Seem
August 16, 2024
Polls got you worried? "Working as intended," some may say.
Most polls, afterall, are media creations that help them generate headlines.
Skewed Sampling
ActiVote: Harris +5. "Harris leaps ahead of Trump." Hard to argue with that headline when one candidate opens up a 5% lead over the former frontrunner. But that's weird... when you look at the independent vote, it splits almost 50-50. And neither candidate is really bleeding more voters from their party compared to the other...
What many observers of this poll fall prey to is the most common way a poll can mislead its audience: a skewed sample. Looking at the "weighted" column for the party affiliation, we see a split of 40% Democrats, 35% Republicans, and 25% Independents. A D+5 lead.
In 2016, exit polls reported that the electorate was composed of 36% Democrats, 33% Republicans, and 31% Independents -- D+3. In 2020, exit polls reported that the electorate was composed of 37% Democrats, 36% Republicans, and 26% Independents -- D+1. We have to go all the way back to 2012 to find the last time Democrats surpassed a turnout edge of this magnitude; in that election, they outnumbered Republicans by 6%. That election was, of course, between our first black president, who was historically popular among his party, and an average Republican who never had his base fully rallied behind him.
The fact remains that Republicans have never outnumbered Democrats in any presidential election this century.1 The closest they came was in 2004 when they tied with Democrats -- 37% each. This fall, Democrats will almost certainly outnumber Republicans, but between the past two elections, Democrats have averaged a 2% edge, not 5. Don't underestimate how much of an impact that makes. Moving from a 5% edge to a 2% takes the poll from a margin of 5.2% down to 2.8%. A 2.8% lead in the national vote for Harris would still translate, according to my model, a win for Trump a little less than 50% of the time. Remember: 2.8% is not far off from the margin that Clinton won the national vote in 2016. If the election comes down to a 1% turnout edge for Democrats, like in 2020, then Harris' lead falls to just over 2%, even less than Clinton's.
Bear in mind, this is with ActiVote's results showing independents splitting 50-50, but if independents go this way or that, then the election can quickly fall out of reach for one of the candidates.
Sudden Methodology Change
Emerson released a poll this week showing a Harris +4 race. That stands in stark contrast to their last poll back in July that showed Trump +4. No doubt the race for the White House has changed since Harris took up the mantle as the Democratic nominee, but such a rapid change in a month's time seems unfathomable. And it might just be.
Tucked away in Emerson's methodology section -- something that doesn't write headlines -- is a breakdown of how voters were contacted for this survey. The three main methods for contacting prospective survey participants are by online, landline, and mobile phone. Think for me for a second here. If you and I conducted a poll exclusively conducted over landline, would you expect that poll result to show an exceptionally pro-conservative result? That is, in fact, exactly what happens. The opposite can be said for online polling. It skews towards Democratic-favorable results, albeit not as dramatically as the landline bias.
Knowing this, as a pollster should, you would expect a pollster to find a reasonably appropriate balance of landline, online, and mobile phone in order to produce polls that are reasonably reflective of the national temperature. And Emerson did... until they abandoned it in this most recent poll.
Emerson has conducted 5 (now 6) national polls since April. In two of them, Emerson only used landline and online, so we'll set those aside. In the other three, like the one they just released, Emerson used all three methods. Between those three polls, Emerson had an average breakdown of 49% online, 28% landline, and 24% mobile phone. In all three surveys, landline never dropped below 20% and online never rose above 55%. Why then all of the sudden did Emerson decide to make a radical shift in August? Suddenly, landline dropped all the way down to 10% and online rose to 67%. It comes as no surprise that this poll produced their most liberal result of this cycle, irrespective of the Harris surge.
Erroneous Weighting
Polls can also go awry with improper weighting as well. Take, New York Times & Siena College's recent Pennsylvania poll that boasted a 4-point Kamala Harris lead in the head-to-head polling. Pennsylvania as a state is closely divided between Republican and Democrat turnout during presidential election years. In 2016, Democrats were 40% of those who turned out while Republicans were 38%. In 2020, Democrats were still 40%, but Republicans rose to 41%. So, in a state this closely divided, the race turns on whoever wins the state's independents.
This early August poll shows Kamala Harris doing just that, winning independents by 8% in the head-to-head and by 9% when all of the candidates are included. Quite troubling numbers if you're in the Trump camp, until you notice something that really stands out.
Another category that survey respondents were asked to cast themselves into was their 2020 vote with the options being "Trump," "Biden," and "Did Not Vote." Trump and Harris each retain 94% and 95% of their party's 2020 vote while each losing 3% to the other. Among the "Did Not Vote" respondents, each took 43%. A statistical tie is certainly a far cry from a 4% Harris lead, even with considering the fact that Trump lost Pennsylvania by 1% in 2020, meaning there would naturally be slightly more "Biden" respondents than "Trump" respondents for the 2020 question.
What's worse for Harris is the full field, where Trump retains a (slightly) higher level of his 2020 vote and also opens up a 3% lead among those who did not vote in 2020.
But none of the headlines wrote about that. They led with the alleged 4% Harris lead.
What this all speaks to is an oversampling of Biden voters from 2020. NYT and Siena College should have seen that and probably did.
Closing
What the news agencies do not tell you is that most of these polls are commissioned by them in order to write news headlines, and it should come as no surprise that these same left-wing news agencies would want polling outcomes that rally their supporters and discourage their opposition.
Don't be fooled, however. There are conservative pollsters like Rasmussen that do the exact same thing but in reverse. They were remarkably accurate in 2016, but have since then been nothing more than an offset for liberally-skewed polls.2
All of this to say: Don't focus too much on the polls. They are random samples, not actual election results. If you want a candidate to win, do what you can within your own capacity to make that happen.