Putting a right wing poll of Washington State under the microscope: An evaluation of Echelon/Future42’s spring 2023 survey

This week, the right wing group Future42 — a “capac­i­ty group” of the right wing polit­i­cal infra­struc­ture ini­tia­tive Project 42 formed not long ago by for­mer Wash­ing­ton Pol­i­cy Cen­ter CEO Dann Mead Smith — released the results of a statewide poll it com­mis­sioned from Ech­e­lon Insights on a range of cur­rent issues.

Right wing com­men­ta­tors have been eager­ly shar­ing screen­shots from this sur­vey on their Twit­ter accounts, while for­mer Q13 Fox per­son­al­i­ty Bran­di Kruse devot­ed a spe­cial episode of her pod­cast to high­light­ing a num­ber of the findings.

Often­times, when a sur­vey like this comes out from a right wing out­fit, the spon­sor sim­ply does­n’t pub­lish enough infor­ma­tion for our team to con­duct an in-depth eval­u­a­tion of the sur­vey. How­ev­er, to their cred­it, Future42 has pub­lished what appears to be their full script and com­plete topline results, along with a descrip­tion of their method­ol­o­gy. That impressed our team, so we decid­ed to take the time to go through the sur­vey and eval­u­ate their work.

Here are our impres­sions of their survey.

How we evaluate polls

First, a few words about how we eval­u­ate polls.

We believe that sub­jec­tive orga­ni­za­tions are per­fect­ly capa­ble of con­duct­ing objec­tive research, so we won’t reflex­ive­ly dis­miss data from a sur­vey mere­ly because it comes from right wing spon­sor. How­ev­er, we are stick­lers for the sci­en­tif­ic method, so we will look to see if that spon­sor and their poll­ster adhered to best prac­tices for obtain­ing sound data, or whether they cut corners.

The key to accu­rate, cred­i­ble polling is neu­tral ques­tions asked of a rep­re­sen­ta­tive sam­ple. You’ve got to have both, or your data is going to be skewed. You can’t find out what peo­ple think if you tell them what to think first, which is why neu­tral ques­tions are impor­tant. And your sam­ple needs to prop­er­ly resem­ble the elec­torate or the pop­u­la­tion of the area you’re sur­vey­ing, or the find­ings sim­ply aren’t going to accu­rate­ly reflect what pub­lic opin­ion is.

It’s extreme­ly dif­fi­cult to write neu­tral ques­tions and it can also be tough to build rep­re­sen­ta­tive sam­ples. But it can be done, as we’ve demon­strat­ed over the past decade in our work with Pub­lic Pol­i­cy Polling of North Car­oli­na, a well-regard­ed poll­ster, and more recent­ly also with Change Research of California.

About the Echelon/Future42 survey

Accord­ing to Ech­e­lon Insights, this was a sur­vey of 600 reg­is­tered vot­ers in the “Wash­ing­ton like­ly elec­torate” (mean­ing, it appears to be a poll of like­ly vot­ers, per­haps like­ly 2024 gen­er­al elec­tion vot­ers, though this is not stat­ed). 70% par­tic­i­pat­ed online (text-to-web) and 30% by land­line (live inter­views). The poll field­ed from June 5th — 7th, 2023, and it has a mar­gin of error of +/- 5.0%.

Ech­e­lon Insights has not released enough pub­lic polling to receive a let­ter grade from FiveThir­tyEight, one of the few enti­ties that rates poll­sters. Instead, it has a pro­vi­sion­al A/B rat­ing. You can see its pro­file on FiveThir­tyEight here.

The firm is based in Vir­ginia, and is Republican-aligned.

(As you can prob­a­bly guess, a Repub­li­can-aligned firm is par­ti­san and works main­ly with right wing clients. A Demo­c­ra­t­ic-aligned firm like either of our poll­sters is the equiv­a­lent, also par­ti­san and works with pro­gres­sive clients.)

A copy of the results file is below.

Evaluating the sample

On sev­er­al fronts, Ech­e­lon Insights and Future42 did alright with their statewide sam­ple. How­ev­er, in cru­cial areas, we think their sam­ple is skewed.

Here’s a score­card of their demographics:

Let’s dive into the prob­lem­at­ic areas.

Ech­e­lon reports that 24% of their respon­dents ini­tial­ly iden­ti­fied as Repub­li­can, 33% Demo­c­ra­t­ic, 35% inde­pen­dent, 6% “some­thing else,” and 2% don’t know / refused / unsure (Ques­tion 27). The ini­tial per­cent­ages for Repub­li­cans and inde­pen­dents seem alright, but the per­cent­age of Demo­c­ra­t­ic vot­ers is too low.

Look­ing at the answers to the very next ques­tion (Ques­tion 28), we can see that the “Some­thing else / don’t know / and refused” vot­ers are more Repub­li­can than Demo­c­ra­t­ic, which means that the sam­ple skews Republican:

QUESTION: If inde­pen­dent, mem­ber of anoth­er par­ty, or unsure, ask: Which of the fol­low­ing state­ments best describes you?

[Ran­dom­ize scale]

ANSWERS:

Total votes for Repub­li­cans: 38%
Total votes for Democ­rats: 27%

1. I almost always vote for Repub­li­can can­di­dates: 15%
2. I vote for Repub­li­cans more often than I vote for Democ­rats: 23%
3. I vote for Democ­rats more often than I vote for Repub­li­cans: 17%
4. I almost always vote for Demo­c­ra­t­ic can­di­dates: 10%
5. I vote equal­ly for Democ­rats and Repub­li­cans: 28%
6. Don’t know/Refused/Unsure: 7%

CONSOLIDATED ANSWERS:

QPar­ty + QPartyLean

1. Total Republican+Republican-leaning Inde­pen­dents: 40%
2. Total Democratic+Democratic-leaning Inde­pen­dents: 46%
3. Total Inde­pen­dent: 13%

Then, in the answers to the very next ques­tion, we get con­fir­ma­tion of the skew: 34% say they are very/somewhat con­ser­v­a­tive and 35% say very/somewhat lib­er­al. The num­ber who are con­ser­v­a­tive are sta­tis­ti­cal­ly the same as the num­ber who are lib­er­al? That sim­ply does­n’t reflect Wash­ing­ton State’s elec­toral profile.

As any­one famil­iar with our state’s pol­i­tics knows, Wash­ing­ton is a pre­dom­i­nant­ly Demo­c­ra­t­ic state. We haven’t vot­ed for a Repub­li­can for pres­i­dent since 1984, a Repub­li­can for gov­er­nor since 1980, or a Repub­li­can for U.S. Sen­ate since 1994… and so on. Democ­rats cur­rent­ly hold every statewide office, have large majori­ties in both cham­bers of the Leg­is­la­ture, and have eight of ten U.S. House seats.

Most vot­ers here are Demo­c­ra­t­ic or lean Demo­c­ra­t­ic. Under­rep­re­sent­ing them in a sur­vey is a car­di­nal sin if you’re doing pub­lic opin­ion research in Washington.

In the edu­ca­tion crosstabs, there’s also a skew.

Those with only a high school edu­ca­tion or less are slight­ly over-rep­re­sent­ed, while col­lege grad­u­ates are slight­ly under­rep­re­sent­ed. Here’s a breakdown:

QUESTION: What is the high­est lev­el of edu­ca­tion you have completed?

ANSWERS:

1. High school or less: 25% [Per­cent­age is too high]
2. Some col­lege: 22% [A lit­tle low]
3. Associate’s degree: 12% [A lit­tle low]
4. Bachelor’s degree: 25% [A lit­tle low]
5. Grad­u­ate degree: 15% [Per­cent­age OK]

Edu­ca­tion nowa­days has a strong cor­re­la­tion with par­ty — the more edu­cat­ed a vot­er is, the more like­ly they are to be a Demo­c­ra­t­ic vot­er ver­sus a Repub­li­can vot­er — so this is addi­tion­al con­fir­ma­tion that their sam­ple skews Republican.

On age, we see more younger vot­ers than we’d expect to see and few­er mid­dle-aged vot­ers. How­ev­er, this is more of a minor quib­ble com­pared to the oth­er two cat­e­gories and prob­a­bly does­n’t neg­a­tive­ly impact the toplines as much.

The bot­tom line? This unfor­tu­nate­ly isn’t a prop­er­ly rep­re­sen­ta­tive sam­ple of the Wash­ing­ton State elec­torate. It’s just too right wing.

The questions

The sur­vey con­tains twen­ty-two top­i­cal or issue ques­tions, grouped under sev­er­al cat­e­gories: Land­scape, The Leg­is­la­ture, Crime/Policing, Cost of Liv­ing and Tax­es, Ener­gy Costs, Housing/Homelessness, and Parental Rights.

Cri­tiquing every ques­tion would result in an extreme­ly lengthy post, so rather than do that, I’ll dis­cuss a few select­ed ques­tions. First, let’s con­sid­er a ques­tion that gets a pass­ing grade with respect to the neu­tral­i­ty of its wording:

QUESTION: If the elec­tion for the Wash­ing­ton state leg­is­la­ture in your dis­trict were held today, would you vote for the [ROTATE: Republican/Democratic] can­di­date or the [ROTATE: Democratic/Republican] candidate?

[PHONE: If Republican/Democratic: And would you DEFINITELY vote for the (Republican/Democratic) can­di­date or only PROBABLY vote for the (Republican/Democratic) candidate?]

ANSWERS:

The Repub­li­can can­di­date, definitely/probably: 44%
The Demo­c­ra­t­ic can­di­date, definitely/probably: 45%

1. The Repub­li­can can­di­date, def­i­nite­ly: 29%
2. The Repub­li­can can­di­date, prob­a­bly: 15%
3. The Demo­c­ra­t­ic can­di­date, prob­a­bly: 18%
4. The Demo­c­ra­t­ic can­di­date, def­i­nite­ly: 27%
5. Don’t know/Refused/Unsure: 11%

The word­ing here works because no attempt is being made to influ­ence peo­ple’s respons­es one way or the oth­er, which is extreme­ly important.

They also rotat­ed their answer options, which they did­n’t have to do, but which is very defen­si­ble. And unless some­thing was omit­ted from the script, they avoid­ed try­ing to stack the deck by putting neg­a­tive mes­sag­ing into pre­vi­ous ques­tions (e.g. by say­ing Demo­c­ra­t­ic leg­is­la­tors have been pass­ing bad policies).

How­ev­er, because their sam­ple skews Repub­li­can, the respons­es to this ques­tion don’t reflect the state’s actu­al elec­toral dynam­ics. We’d expect to see some­where in the range of like a five to ten point lead for the Demo­c­ra­t­ic Par­ty in a gener­ic bal­lot ques­tion like this. Instead, the Repub­li­cans are even with the Democrats.

This is an unsup­port­able finding.

Sim­i­lar­ly, in an adja­cent ques­tion, imme­di­ate­ly pri­or to this one, respon­dents were asked whether they have a favor­able or unfa­vor­able view of three peo­ple and two groups: Com­mis­sion­er of Pub­lic Lands Hilary Franz, Democ­rats in the Wash­ing­ton State Leg­is­la­ture, Attor­ney Gen­er­al Bob Fer­gu­son, Gov­er­nor Jay Inslee, and Repub­li­cans in the Wash­ing­ton State Legislature.

The ques­tion word­ing looks just fine.

But because of the sam­ple’s Repub­li­can skew, the favor­a­bil­i­ty rat­ings don’t line up with what we’d see in a cred­i­ble pub­lic opin­ion research survey.

For exam­ple, con­sid­er Gov­er­nor Inslee’s favor­a­bil­i­ty rating:

QUESTION: Do you have a favor­able or unfa­vor­able view of the fol­low­ing groups or indi­vid­u­als in the news late­ly? If you haven’t heard of this per­son, just say so. [RANDOMIZE OPTIONS] [PHONE: If favorable/unfavorable: And would you say you have a VERY (favorable/unfavorable) view or only a SOMEWHAT (favorable/unfavorable) view?

ANSWERS FOR GOVERNOR JAY INSLEE:

• Favor­able (Very/somewhat): 40% 
  • Very favor­able: 20%
  • Some­what favor­able: 20%
• Unfa­vor­able (Very/somewhat): 56% 
  • Some­what unfa­vor­able: 12%
  • Very unfa­vor­able: 44%
• Heard of / no opin­ion: 3%
• Haven’t heard of: 1%

We peri­od­i­cal­ly ask about Gov­er­nor Jay Inslee’s job per­for­mance, includ­ing as recent­ly as last month. That’s not the same thing as favor­a­bil­i­ty, but it is a sim­i­lar line of inquiry. Typ­i­cal­ly, we see close to half our sam­ple giv­ing Gov­er­nor Inslee pos­i­tive marks and close to half giv­ing him neg­a­tive marks.

That was true again about a month ago:

QUESTION (FROM NPI’S RESEARCH): Do you approve or dis­ap­prove of Gov­er­nor Jay Inslee’s job performance?

ANSWERS (JUNE OF 2023):

• Approve: 48%
• Dis­ap­prove: 47%
• Not sure: 5%

Our sur­vey of 773 like­ly 2024 Wash­ing­ton State vot­ers was in the field from Wednes­day, June 7th through Thurs­day, June 8th, 2023.

The poll uti­lizes a blend­ed method­ol­o­gy, with auto­mat­ed phone calls to land­lines (41%) and online answers from cell phone only respon­dents (59%).

It was con­duct­ed by Pub­lic Pol­i­cy Polling (PPP) for the North­west Pro­gres­sive Insti­tute, and has a mar­gin of error of +/- 3.5% at the 95% con­fi­dence interval.

NPI and PPP have worked togeth­er for a decade and have a track record of excel­lence, as detailed in this 2022 elec­toral polling recap and this 2020 one.

You can see from oth­er pub­lic opin­ion research since the 2020 pres­i­den­tial and guber­na­to­r­i­al elec­tion that our find­ing is a typ­i­cal one, where­as Echelon/Future42’s is an out­lier. It is not sur­pris­ing that a poll that over-sam­ples Repub­li­cans would yield more neg­a­tive num­bers for Gov­er­nor Inslee, as well as Attor­ney Gen­er­al Bob Fer­gu­son, who’s now run­ning for governor.

Now let’s look at a ques­tion that utter­ly fails the neu­tral­i­ty test.

QUESTION: Under Fed­er­al law, cap­i­tal gains are con­sid­ered income, and the state con­sti­tu­tion of Wash­ing­ton pro­hibits an income tax. Recent­ly, the State Supreme Court ruled that a cap­i­tal gains tax is per­mis­si­ble if it is called an “excise tax”. Was the Supreme Court right or wrong to revise the def­i­n­i­tion of income to allow for this new tax?

(Answer options were ran­dom­ized)

ANSWERS:

• Wrong: 62%
• Right: 24%
• Don’t know / refused / unsure: 14%

This is def­i­nite­ly a con­tender for the worst ques­tion in this poll: it’s prej­u­di­cial­ly word­ed and it con­tains inac­cu­rate infor­ma­tion. Let’s cor­rect the record:

1. What fed­er­al law says about cap­i­tal gains is not rel­e­vant because peo­ple are being asked about a state law which levied a cap­i­tal gains tax only on the wealth­i­est Wash­ing­ton State fam­i­lies and households.
2. The Wash­ing­ton State Con­sti­tu­tion does not pro­hib­it an income tax. This is a myth. Go ahead and read our plan of gov­ern­ment for your­self — nowhere will you find a pro­vi­sion say­ing that the state may not impose an income tax. There is an old Supreme Court deci­sion from the 1930s which held that income is con­sid­ered prop­er­ty, and there is a clause in the Con­sti­tu­tion (the “uni­for­mi­ty clause”) that says tax­es on prop­er­ty must be uni­form, but that’s not a pro­hi­bi­tion against hav­ing an income tax.
3. The state Supreme Court’s recent rul­ing in Quinn is egre­gious­ly mis­char­ac­ter­ized here. The Court did not rule that a cap­i­tal gains tax is per­mis­si­ble if it is called an excise tax. Rather, it ruled the cap­i­tal gains tax is per­mis­si­ble because it is an excise tax. The Court exam­ined the tax in-depth and the law that cre­at­ed it, and emphat­i­cal­ly reject­ed the right wing’s argu­ment that it is not real­ly an excise tax.
4. And final­ly, the Supreme Court did not “revise the def­i­n­i­tion of income to allow for this new tax.” Rather, the Court con­clud­ed that the Leg­is­la­ture had act­ed with­in its law­ful author­i­ty to cre­ate an excise tax on cap­i­tal gains. See for your­self by read­ing the major­i­ty opin­ion.

At no point in this ques­tion are respon­dents giv­en any pos­i­tive infor­ma­tion about the cap­i­tal gains tax on the wealthy. At no point are they told who pays it and what’s exempt. Instead, they’re told the tax is bad and then invit­ed to agree with the authors of the ques­tion that the Supreme Court erred by uphold­ing the tax.

As I men­tioned ear­li­er, you can’t find out what peo­ple think if you tell them what to think first. Future42 did not write this ques­tion in a way that would allow the respon­dents to bring their own opin­ions to the table. Instead, they wrote a loaded ques­tion which reads like a Tim Eyman style push poll. Bad ques­tions and a skewed sam­ple com­bine here to yield utter­ly worth­less figures.

Here is what a bal­anced, neu­tral­ly word­ed ques­tion looks like. This is one we’ve asked for sev­er­al con­sec­u­tive years now, includ­ing last month:

QUESTION (FROM NPI’S RESEARCH): Pro­po­nents say that Wash­ing­ton State’s new state cap­i­tal gains tax on the wealthy will raise about $500 mil­lion a year in cru­cial fund­ing for edu­ca­tion in Wash­ing­ton State, includ­ing ear­ly learn­ing and child­care, and will help bal­ance our upside-down tax code by requir­ing the wealth­i­est 8,000 indi­vid­u­als to step up and pay their fair share in dues to our state. Oppo­nents say that the new state cap­i­tal gains tax on the wealthy is an uncon­sti­tu­tion­al and ille­gal income tax that will hurt job cre­ation and put the state at a com­pet­i­tive dis­ad­van­tage, hurt­ing the whole econ­o­my while fail­ing to address regres­siv­i­ty. Both sides agree that the text of the cap­i­tal gains tax law ful­ly exempts retire­ment accounts, fam­i­ly farms, and all real estate. Hav­ing heard the argu­ments for and against, do you strong­ly sup­port, some­what sup­port, some­what oppose, or strong­ly oppose Washington’s new state cap­i­tal gains tax on the wealthy?

ANSWERS (JUNE OF 2023):

• Sup­port: 57% 
  • Strong­ly sup­port: 44%
  • Some­what sup­port: 13%
• Oppose: 37%
  • Some­what oppose: 9%
  • Strong­ly oppose: 28%
• Not sure: 6%

Sup­port is about the same as when we asked last year (57% now ver­sus 56% last June.) Oppo­si­tion is the same, at 37%, with a tiny, sta­tis­ti­cal­ly insignif­i­cant bit of soft­en­ing. The num­ber of not sure vot­ers has also like­wise bare­ly changed.

You’ll notice that in our ques­tion, the uncon­sti­tu­tion­al and ille­gal income tax fram­ing is present, but it’s appro­pri­ate­ly attrib­uted to oppo­nents of the tax rather than being passed off as a fact. (If you’re curi­ous, the “con” phras­ing used came direct­ly from a group led by Rob McKen­na — Oppor­tu­ni­ty Washington.)

The qual­i­ty of the non-demo­graph­ic ques­tions real­ly var­ied in this poll.

Gen­er­al­ly speak­ing, the elec­toral ques­tions and favor­a­bil­i­ty ques­tions had accept­able word­ing, while the ques­tions about issues uti­lized right wing fram­ing and lacked neu­tral­i­ty to vary­ing degrees. Inter­est­ing­ly, Ech­e­lon gen­er­al­ly asked respon­dents whether some­thing was “a good idea” or “a bad idea” rather than ask­ing if they sup­port­ed or opposed it, which is the dichoto­my we gen­er­al­ly use. Their results file says the answer options were ran­dom­ized as a gen­er­al practice.

Concluding thoughts

Our assess­ment is that this sur­vey skews Repub­li­can by as much as ten points, which is coin­ci­den­tal­ly twice its stat­ed mar­gin of error. Accord­ing­ly, even the topline respons­es to the neu­tral­ly word­ed ques­tions are flawed.

That’s dis­ap­point­ing, because this sur­vey might oth­er­wise have yield­ed some use­ful find­ings and added to our state’s body of cred­i­ble pub­lic opin­ion research.

For their next sur­vey, we urge Echelon/Future42 to (1) fix their sam­pling to include a suf­fi­cient num­ber of Demo­c­ra­t­ic and col­lege-edu­cat­ed vot­ers and (2) sim­pli­fy their par­ty iden­ti­fi­ca­tion ques­tion, which also could help with (1). Those moves might at least yield usable data from their elec­toral ques­tions, even if their issue-relat­ed ques­tions remain poor­ly and prej­u­di­cial­ly written.