Wednesday, June 12, 2024

This poll is so bad, we’ve revived the saltshaker test for it. The whole shaker.

Read on blog or Reader

This poll is so bad, we've revived the saltshaker test for it. The whole shaker.

Steve Vancore

June 12

Oh Canada!

If you are reading the latest poll from FAU/Mainstreet Research, STOP!

It seems compelling, with apparent insights into the presidential race, including how former President Donald Trump and current President Joe Biden would perform in a head-to-head match-up and in a race with independent and third party candidates. In a good poll, that would give readers some indication of which candidate the spoilers are spoiling.

The poll also surveys respondents on U.S. Sen. Rick Scott's reelection, and the Democratic Primary to challenge him (though it's Alan Grayson in the poll and as we now know, he's not running in that race anymore.) It also looks forward at possible gubernatorial candidates to replace Gov. Ron Desantis, including his wife, Casey DeSantis.

So, yes, it's a tempting read. But again, STOP!

Several years ago, we stopped writing our "Saltshaker" analysis of polls for a variety of reasons, but the latest poll from FAU screams for correction. So, let's break out the 2024 version of the saltshaker test!

This poll is so poorly conceived and constructed that frankly it embarrasses — or at least it should embarrass — an otherwise fine Florida-based institution of higher learning.

How bad is this poll? Let us count just some of the ways…

It surveys "adults"

Any election poll that begins with "among a sample of X adults" should be immediately discarded for several reasons. First, in Florida only registered voters can vote. Seems obvious, doesn't it? (Maybe not so much to a Canadian-based polling firm.) When you consider that only about 80% (or so) of adults are registered to vote and, at most, about 80% of them will vote in November, back of the napkin math shows that about 36% of adults will not vote in November. So more than a third of your sample is essentially a non-voter. Why bother talking to those who can't or won't vote? Second, if you don't use the voter file to identify and find actual active voters (which they apparently did not), then you must rely on self-description of how individuals are registered. For a variety of reasons (people forget, people don't want to disclose, people wish they were a different party etc.), this also leads to skewed results.

Party balance

Ugh! The survey had more Democrats than Republicans! All Florida pollsters agree that given current voter registration trends, coupled with past turnout trends, we should see what we call a "+7 or +8 GOP" model. What that means is that on Election Day the final tallies should end up with about seven or eight percentage points more Republicans than Democrats. Democratic optimists might be shooting for a +5 or +6, but no serious operative thinks there will be more Democrats than Republicans voting.

Party balance, Part Deux

Also, ugh! The overall balance of the poll is ridiculously out of balance with more Democrats than Republicans polled so the pollster re-weighted the sample in a way that is "intended to represent the voting population in Florida." That could be kind of OK if done correctly, except it was not done correctly. Instead, the pollsters re-weighted the sample (post-survey re-weighting has problems in and of itself) but they didn't make it look like the likely voting electorate at all. They re-weighted it to show a +2 GOP model. As noted above, not even optimistic Democrats think that will happen because there isn't a shred of data that suggests it will. To further make the point, their newly weighted sample has only 19% "independent" voters when in reality it should be around 26%-28% "NPA/Minor Party" voters.

The bottom line

For readers of this site, we presume you are not the typical audience and are more politically active and engaged. With that, we would like to offer a word of caution when reading polls released by either a media outlet or a university. Most of these polls are designed to generate clicks and therefore are generally done on the cheap meaning they simply must cut corners.

If you are hiring McLaughlin, Tyson, Eldon or some of the other well-established pollsters (like our own Clearview Research) who know and understand Florida, we can't afford to cut those corners because we wouldn't last in this business. A good poll costs money and a good poll requires (more than ever) a strict adherence to proven methodologies. In the current circumstance, this simply is not a good poll.

Sorry FAU and Mainstreet you get the whole shaker of salt.