Poll Watch: Thoughts on the New York Times / Siena College’s battleground state polling

In part­ner­ship with Siena Col­lege, The New York Times recent­ly sur­veyed vot­ers in six key swing states about the 2024 pres­i­den­tial elec­tion, which is one year away. (Gen­er­al Elec­tion Day in 2024 will be on Tues­day, Novem­ber 5th).

The NYT and Siena found Don­ald Trump — who has been charged with incit­ing an insur­rec­tion against the Unit­ed States — ahead of Pres­i­dent Joe Biden in five out of the six states, with Biden hold­ing onto a nar­row lead only in Wisconsin:

The results show Mr. Biden los­ing to Mr. Trump, his like­li­est Repub­li­can rival, by mar­gins of four to ten per­cent­age points among reg­is­tered vot­ers in Ari­zona, Geor­gia, Michi­gan, Neva­da and Penn­syl­va­nia. Mr. Biden is ahead only in Wis­con­sin, by two per­cent­age points, the poll found.

These are state-lev­el polls in key swing states where the 2024 pres­i­den­tial elec­tion is expect­ed to be decid­ed, so they are worth scru­ti­niz­ing, unlike the nation­al polls that don’t offer a whole lot of val­ue. Here are some thoughts from our team after review­ing the poll results that the NYT pub­lished. (We were much more inter­est­ed in the data they shared than their take on it.)

Republican voters were oversampled in these polls

The word “over­sam­pled” does­n’t appear in either Shane Gold­macher’s write­up of the poll results or Nate Cohn’s write­up. How­ev­er, if you open the actu­al dataset, look through it, and read the end­notes, a key fact should become imme­di­ate­ly appar­ent: Repub­li­can vot­ers were over­sam­pled in these polls.

It’s irre­spon­si­ble of the NYT to not state this upfront in all of its cov­er­age. This is an impor­tant design choice that they made, and it ought to be explained.

The New York Times and its part­ners had to do a bunch of weight­ing because of their deci­sion to do over­sam­pling. Here is their expla­na­tion, which is only in the end­notes, not in any of the sto­ries they’ve writ­ten so far about the poll. It’s long:

Weight­ing — reg­is­tered voters

The sur­vey was weight­ed by The Times using the R sur­vey pack­age in mul­ti­ple steps to account for the over­sam­ple of Repub­li­can voters.

First, the sam­ples were adjust­ed for unequal prob­a­bil­i­ty of selec­tion by stratum.

Sec­ond, the six state sam­ples were weight­ed sep­a­rate­ly to match vot­er file-based para­me­ters for the char­ac­ter­is­tics of reg­is­tered vot­ers by state.

The fol­low­ing tar­gets were used:

• Par­ty (par­ty reg­is­tra­tion if avail­able, else clas­si­fi­ca­tion based on a mod­el of vote choice in pri­or Times/Siena polls)
• Age (Self-report­ed age, or vot­er file age if the respon­dent refuses)
• Gen­der (L2 data)
• Race or eth­nic­i­ty (L2 model)
• Edu­ca­tion (four cat­e­gories of self-report­ed edu­ca­tion, weight­ed to match NYT-based tar­gets derived from Times/Siena polls, cen­sus data and the L2 vot­er file)
• Mar­i­tal sta­tus (L2 model)
• Home own­er­ship (L2 model)
• State regions (NYT clas­si­fi­ca­tions by coun­ty or city)
• Turnout his­to­ry (NYT clas­si­fi­ca­tions based on L2 data)
• Vote method in the 2020 elec­tions (NYT clas­si­fi­ca­tions based on L2 data)
• Cen­sus block group den­si­ty (A.C.S. 5‑Year Cen­sus Block Group data)
• City type (Neva­da only, added based on a post-hoc analy­sis of the dif­fer­ence between the weight­ed sam­ple and vot­er file para­me­ters. The weight had no mean­ing­ful effect on the topline result.)
• Cen­sus tract edu­ca­tion­al attain­ment (Geor­gia only, added based on a post-hoc analy­sis of the dif­fer­ence between the weight­ed sam­ple and vot­er file para­me­ters. The weight had no mean­ing­ful effect on the topline result.)

Final­ly, the six state sam­ples were bal­anced to each rep­re­sent one-sixth of the sum of the weights.

Weight­ing — like­ly electorate

The sur­vey was weight­ed by The Times using the R sur­vey pack­age in mul­ti­ple steps to account for the over­sam­ple of Repub­li­can voters.

First, the sam­ples were adjust­ed for unequal prob­a­bil­i­ty of selec­tion by stratum.

Sec­ond, the first-stage weight was adjust­ed to account for the prob­a­bil­i­ty that a reg­is­trant would vote in the 2024 elec­tion, based on a mod­el of turnout in the 2020 election.

Third, the six state sam­ples were weight­ed sep­a­rate­ly to match tar­gets for the com­po­si­tion of the like­ly elec­torate. The tar­gets for the com­po­si­tion of the like­ly elec­torate were derived by aggre­gat­ing the indi­vid­ual-lev­el turnout esti­mates described in the pre­vi­ous step for reg­is­trants on the L2 vot­er file. The cat­e­gories used in weight­ing were the same as those pre­vi­ous­ly men­tioned for reg­is­tered voters.

Fourth, the ini­tial like­ly elec­torate weight was adjust­ed to incor­po­rate self-report­ed vote inten­tion. The final prob­a­bil­i­ty that a reg­is­trant would vote in the 2024 elec­tion was four-fifths based on their ex ante mod­eled turnout score and one-fifth based on their self-report­ed inten­tion, based on pri­or Times/Siena polls, includ­ing a penal­ty to account for the ten­den­cy of sur­vey respon­dents to turn out at high­er rates than non­re­spon­dents. The final like­ly elec­torate weight was equal to the mod­eled elec­torate rake weight, mul­ti­plied by the final turnout prob­a­bil­i­ty and divid­ed by the ex ante mod­eled turnout probability.

Final­ly, the six state sam­ples were bal­anced to each rep­re­sent one-sixth of the sum of the weights.

The NPI team did not see an expla­na­tion giv­en for why Repub­li­can vot­ers were over­sam­pled, but we can guess why. It is under­stand­able that the New York Times and Siena don’t want to make the mis­take of under­es­ti­mat­ing Trump’s sup­port, giv­en some of the issues that have plagued polling in pre­vi­ous cycles.

If one of the news­pa­per’s research objec­tives was to do a deep dive into Repub­li­can vot­ers’ views on the 2024 pres­i­den­tial race and key issues, then it would have made more sense, in our view, to do a sep­a­rate research project focus­ing exclu­sive­ly on Repub­li­can vot­ers, even if would cost more.

Just how few progressive or partially progressive voters are in the sample? The percentages may surprise you

Most of the NYT’s cov­er­age so far has focused on how Demo­c­ra­t­ic and Demo­c­ra­t­ic lean­ing respon­dents feel about Joe Biden’s reelec­tion, as opposed to how Repub­li­can vot­ers feel about Don­ald Trump or anoth­er Repub­li­can seek­ing the nom­i­na­tion, such as Ron DeSan­tis or Nik­ki Haley.

The crosstabs for these polls show that 75% of the respon­dents iden­ti­fy as very con­ser­v­a­tive, some­what con­ser­v­a­tive, or “mod­er­ate” (which isn’t an ideology):

QUESTION: Do you con­sid­er your­self polit­i­cal­ly lib­er­al, mod­er­ate, or conservative?

FOLLOW UP: (If lib­er­al or con­ser­v­a­tive) Is that very or somewhat?

ANSWERS [ALL STATES]:

• Very lib­er­al: 10%
• Some­what lib­er­al: 10%
• Mod­er­ate: 39%
• Some­what con­ser­v­a­tive: 19%
• Very con­ser­v­a­tive: 17%
• Don’t know/Refused: 5%

Poll­sters can’t know who is actu­al­ly going to show up and vote next year, so they have to con­struct mod­els to project what the uni­verse of par­tic­i­pat­ing vot­ers might be. And, as men­tioned above, Repub­li­can vot­ers were over­sam­pled — on pur­pose. This required the NYT and its part­ners to do a bunch of weighting.

A whop­ping four-fifths of the total sam­ple said they con­sid­er them­selves some­thing oth­er than lib­er­al, or refused to answer. 39% iden­ti­fied as “mod­er­ate” — a label that sounds good but does­n’t stand for any­thing — and 36% iden­ti­fied as con­ser­v­a­tive, near­ly twice as many as those who said liberal.

Now, here are the par­ty affil­i­a­tion numbers:

QUESTION: Do you con­sid­er your­self a Demo­c­rat, a Repub­li­can, an inde­pen­dent or a mem­ber of anoth­er party?

ANSWERS [ALL STATES]:

• Democra: 30%
• Repub­li­can: 32%
• Inde­pen­dent: 30%
• Anoth­er part: 4%
• Don’t know / refused: 5%

FOLLOW-UP QUESTION: (If inde­pen­dent, anoth­er par­ty, or don’t know on par­ti­san­ship) And as of today, do you lean more to:

ANSWERS [ALL STATES]:

• The Demo­c­ra­t­ic Par­ty: 38%
• The Repub­li­can Par­ty: 44%
• Don’t know / refused: 18%

COMBINED ANSWERS: Par­ty iden­ti­fi­ca­tion and leaners

• The Demo­c­ra­t­ic Par­ty: 45%
• The Repub­li­can Par­ty: 49%
• Don’t know/Refused: 7%

The deci­sion to over­sam­ple Repub­li­cans and use weight­ing to account for that result­ed in a more com­plex pub­lic opin­ion research project — one that is hard­er to eval­u­ate and scru­ti­nize. The NYT’s reporters and ana­lysts should have stat­ed upfront in their analy­sis that Repub­li­cans were oversampled.

They did­n’t. That’s unacceptable.

Penn­syl­va­nia, Neva­da, Geor­gia, Ari­zona, Wis­con­sin, and Michi­gan are all swing states where either par­ty can win. Recent elec­tions — in 2022, 2020, 2018, and 2016 — have demon­strat­ed this. The New York Times’ news­room and data team thought Repub­li­cans would be very dom­i­nant in the midterms, as we doc­u­ment­ed right here on The Cas­ca­dia Advo­cate a lit­tle over a year ago.

But the red wave that the NYT’s peo­ple thought was com­ing did not mate­ri­al­ize. Repub­li­cans did just well enough to get a slim major­i­ty in the U.S. House but failed to flip the Sen­ate. Repub­li­cans lost guber­na­to­r­i­al races in four of the six swing states men­tioned above (Penn­syl­va­nia, Ari­zona, Wis­con­sin, and Michi­gan) as well as con­trol of the Leg­is­la­ture in Michi­gan and the Penn­syl­va­nia House.

It should be not­ed that the New York Times and Siena Col­lege have a decent rep­u­ta­tion. For instance, their WA-08 polling in 2018 was on the mark, show­ing slight leads for Demo­c­ra­t­ic hope­ful Kim Schri­er (who won), while local firm Elway Research got it wrong, find­ing a sig­nif­i­cant lead for Dino Rossi (who lost).

But even if this polling is on the mark and does reflect the cur­rent dynam­ics, it’s vital to remem­ber the old say­ing that polls are snap­shots in time. They are not pre­dic­tive, and can’t be. At best, they can be sug­ges­tive, but the data can only be use­ful if the ques­tions are neu­tral and the sam­ples prop­er­ly representative.

We can’t know if the weight­ed sam­ples in these bat­tle­ground state polls reflect the elec­torate that’ll turn out next year. Our team does­n’t think it’s pos­si­ble to make even an edu­cat­ed guess about who’ll show up at this junc­ture, not even with the best avail­able soft­ware, sources of data, and poll design expertise.

Speak­ing of prob­lem­at­ic media coverage…

The mass media is failing Americans at an incredibly critical juncture for American democracy

Amer­i­ca is a coun­try of over 335 mil­lion peo­ple — the third most pop­u­lous in the world after India and Chi­na. There’s no way that any U.S. pres­i­dent can have per­son­al rela­tion­ships with even a mil­lionth of the peo­ple that they rep­re­sent. What Amer­i­cans see and hear about the Biden-Har­ris admin­is­tra­tion is there­fore deter­mined in large part by the mass media, which nowa­days has very con­cen­trat­ed own­er­ship, mean­ing just a few com­pa­nies own most of the media.

By any objec­tive stan­dard, much of the media is doing a ter­ri­ble, awful, hor­ri­ble job of cov­er­ing pol­i­tics and keep­ing Amer­i­cans informed about what real­ly mat­ters. Folks like James Fal­lows and Jay Rosen reg­u­lar­ly point this out. Polit­i­cal cov­er­age has become very shal­low and rat­ings dri­ven, writ­ten by reporters look­ing pri­mar­i­ly through the prism of who’s win­ning and who’s los­ing. Sen­sa­tion­al­ism is in, sub­stance is out — and it’s been that way for a long time.

We can see the con­se­quences of the fourth estate’s descent into absur­di­ty in these poll results. Con­sid­er the fol­low­ing set of ques­tions and respons­es from the poll, which ask about Biden and Trump’s policies:

QUESTION: Do you think Joe Biden’s poli­cies have helped you per­son­al­ly or hurt you personally?

ANSWERS [VOTERS IN ALL SWING STATES]:

• Helped: 35%
• Hurt: 53%
• Don’t know/Refused: 12%

INDEPENDENTS ONLY:

• Helped: 32%
• Hurt: 53%
• Don’t know/Refused: 14%

QUESTION: Do you think Don­ald Trump’s poli­cies have helped you per­son­al­ly or hurt you personally?

ANSWERS [VOTERS IN ALL SWING STATES]:

• Helped: 51%
• Hurt: 34%
• Don’t know/Refused: 14%

INDEPENDENTS ONLY:

• Helped: 52%
• Hurt: 32%
• Don’t know/Refused: 16%

53% of respon­dents said Biden’s poli­cies have hurt them. That’s objec­tive­ly non­sen­si­cal: Biden has request­ed from Con­gress and signed into law a long list of bills to help most Amer­i­cans. For exam­ple, the Infla­tion Reduc­tion Act is final­ly allow­ing Medicare to nego­ti­ate the price of pre­scrip­tion drugs. The Amer­i­can Res­cue Plan reduced child pover­ty with the Child Tax Cred­it. The admin­is­tra­tion is using its author­i­ty under exist­ing law to try to for­give stu­dent loan debt.

And so on.

But oth­er polling has showed Amer­i­cans sim­ply don’t know about these things.

In the absence of that knowl­edge, vot­ers are draw­ing con­clu­sions based on what is being said about Biden rather than what Biden has actu­al­ly done.

Mean­while, vot­ers respond­ed favor­ably when asked about Don­ald Trump’s poli­cies, despite those poli­cies hav­ing been incred­i­bly destructive.

Per­cep­tion and real­i­ty have become extreme­ly dis­con­nect­ed. The mass media might not deserve all of the blame for that, but they cer­tain­ly deserve a lot.

“Amer­i­cans have heard the most about the Biden admin­is­tra­tion lowering
pre­scrip­tion drug costs, ramp­ing up clean ener­gy, and invest­ing in infrastructure,
but even on these accom­plish­ments, few­er than three in five Amer­i­cans have
heard about each one,” Nav­gia­tor report­ed last month.

“Com­mu­ni­cat­ing around Biden’s accom­plish­ments improves his net job approval
rat­ing by dou­ble dig­its, espe­cial­ly among peo­ple unfa­vor­able to both Biden and
Trump, younger Amer­i­cans, and col­lege-edu­cat­ed women.”

The key take­away: Once vot­ers learn about what Biden and Har­ris have got­ten done, they become more enthu­si­as­tic about sup­port­ing the tick­et in 2024.

Based on what has been pub­lished so far (most of the script was released, but not all of it) the New York Times and its research part­ners appar­ent­ly did­n’t test the pop­u­lar­i­ty of spe­cif­ic poli­cies and then revis­it the ques­tion of whether peo­ple feel Biden’s poli­cies have helped or hurt them. We would have been inter­est­ed in see­ing the effect of vot­ers learn­ing about Biden and Har­ris’ actu­al record.

Here’s anoth­er set of ques­tions and responses:

QUESTION: Do you think Joe Biden has the men­tal sharp­ness to be an effec­tive president?

ANSWERS [VOTERS IN ALL SWING STATES]:

• Yes: 35%
• No: 62%
• Don’t know/Refused: 3%

QUESTION: Do you think Don­ald Trump has the men­tal sharp­ness to be an effec­tive president?

ANSWERS [VOTERS IN ALL SWING STATES]:

• Yes: 54%
• No: 44%
• Don’t know/Refused: 2%

These are ridicu­lous yet actu­al­ly believ­able num­bers giv­en who’s in the sam­ple and con­sid­er­ing what pass­es for polit­i­cal cov­er­age in this country.

It’s also impor­tant to remem­ber that there’s an entire right wing media ecosys­tem in the Unit­ed States and beyond devot­ed to mak­ing Don­ald Trump look good and either ignor­ing or mak­ing excus­es for his un-Amer­i­can, fright­en­ing plat­form of fas­cism. It con­sists of Rupert Mur­doch’s FNC, AM right wing talk radio, pod­casts like the one Steve Ban­non puts out, and web­sites like Bre­it­bart and TownHall.

Biden and Har­ris, on the oth­er hand, don’t have a fawn­ing media appa­ra­tus at their dis­pos­al pro­duc­ing non­stop flat­ter­ing cov­er­age. The small pro­gres­sive media ecosys­tem that Amer­i­ca has does not cov­er Biden and Har­ris the way right wing media fawn­ing­ly and obe­di­ent­ly cov­ers Trump. On any giv­en day, a pro­gres­sive pub­li­ca­tion could be either crit­i­ciz­ing or prais­ing the admin­is­tra­tion — or per­haps even both, depend­ing on what the top­ics being cov­ered are.

For exam­ple, Biden’s deci­sion two years ago to renom­i­nate Jerome Pow­ell to be Fed Chair — a deci­sion that has result­ed in Amer­i­can mon­e­tary pol­i­cy remain­ing in the hands of a Don­ald Trump appointee with diver­gent pri­or­i­ties from those of Biden’s — was (jus­ti­fi­ably) crit­i­cized by many pro­gres­sive media outlets.

Don­ald Trump knows a lot about how the media oper­ates and con­stant­ly uses that knowl­edge to his advan­tage. Trump is at the same time both a per­son and a per­sona cre­at­ed and sus­tained by the mass media. Regard­less of your ide­l­og­i­cal views, you’ve like­ly noticed that Trump the per­son is able to stay stu­pid, false, and harm­ful things all of the time with­out Trump the per­sona tak­ing too many hits.

Any objec­tive analy­sis of what is known about Don­ald Trump, includ­ing reports from for­mer under­lings who observed him direct­ly, can’t sup­port the con­clu­sion that he has the men­tal sharp­ness to be an effec­tive pres­i­dent. Unlike eight years ago, when the pos­si­bil­i­ty of Trump in the Oval Office was still a hypo­thet­i­cal, we have a record of words and deeds to go on. It’s fright­en­ing­ly awful.

Yet, a major­i­ty of vot­ers in this research project think Trump has men­tal sharp­e­ness that Biden lacks. Repub­li­cans were over­sam­pled, but still.

Don’t let polls get you down: reframe if you want to be an effective activist, candidate, or leader who will help save American democracy from fascism in 2024

In Think­ing Points, a hand­book he wrote with the Rock­ridge Insti­tute, pro­gres­sive lin­guist George Lakoff made the impor­tant point that true lead­ers don’t fol­low polls. Rather, they lead peo­ple to new posi­tions, for that is what lead­er­ship is.

That time­less les­son is applic­a­ble here.

Sav­ing Amer­i­can democ­ra­cy from fas­cism in 2024 is achiev­able and worth doing. Any­one who wants to make a pos­i­tive con­tri­bu­tion to the cause must be will­ing to reframe and allo­cate what­ev­er they have to give wisely.

At NPI, we are fond of the max­im that fret­ting is like sit­ting in a rock­ing chair — it gives you some­thing to do, but it does­n’t get you anywhere.

Fig­ure out how to chan­nel any anx­i­ety you are feel­ing after see­ing these polls into some­thing use­ful, and you will have just under­tak­en a very worth­while exer­cise for next year. Prac­tice now so you can be the activist you want to be in 2024.

We have choic­es with respect to how we spend our time, our trea­sure, and our tal­ents. We are not required to think about the 2024 elec­tion using the deeply prob­lem­at­ic frames that the mass media use. We have the free­dom to reframe — and we must use that free­dom so that we can real­ize our col­lec­tive poten­tial in 2024. Refram­ing is the key to get­ting in gear for the pres­i­den­tial election.

If we lead, the cred­i­ble polling will even­tu­al­ly fol­low. The research, media, and mes­sag­ing we need are not going to mag­i­cal­ly appear of their own accord. We must build the nec­es­sary polit­i­cal infra­struc­ture to save Amer­i­can democ­ra­cy and not only deploy it for 2024, but keep it going beyond the elec­tion, for suc­cess will mean more elec­tions in the years to come.

We at NPI will keep doing our part, and we invite you to sup­port us. Make a one time dona­tion here or become a mem­ber — it will make a difference!