Are political polls accurate? CA recall study of 20 polls says...NEVER TRUST POLLS AGAIN!

RealClearPolitics.com & original material ^ | 10/21/03 | Wolfstar

[RatSlayer]

you said
"The study focuses on the accuracy of polls, not on their validity. My underlying premise is that the accuracy of most polls can't be proven."

While I applaud your effort, your study is invalid. To do a valid study of your premise, you would have to study the results of polling in a few hundred elections, or at least a couple of dozen elections, not one.

Your study is an extreme case of cherry picking the data. I'm not alleging it was intentional, just that you happened to pick an election in which the unusual voter turnout almost guaranteed inaccurate polls.

Actually, to avoid any charges of cherry picking the data, you should really identify a couple of thousand elections for which you can get some polling data and the election results. Then let a computer program do a random drawing of the couple of hundred elections which will be your study sample. Only at this point, do you actually go out and collect the data. This way someone cannot accuse you of knowing which elections to include in the random drawing.

Since polls are entirely dependant on the demographic choices made by the pollsters, Obviously, in an election such as the recall, where a large percentage of first time voters cast ballots, the odds of the polls being wrong are greatly increased.

An interesting study would be to determine why the demographics of that one poll yielded good results. But, that would take inside information.

[Marie Antoinette]

Nice! Thanks for all you do.

[Lonesome in Massachussets]

I think your analysis is mathematically flawed. Trouble with independent events and all that. For instance, "YES" and "NO" are not independent. The stated margin of error (MOE) in a poll is purely the 95-percentile sampling error. It is only a statement of the uncertainty attributable to a finite sample size.

The bigger problem with polls is systematic error. People who take time to talk to pollsters (I'm not one of them) are not representative of the population. An honest poll works to suppress systematic error, there are so-called "push polls" designed to produce a certain result where every attempt is made to build in systemic error.

To illustrate MOE: If there are one million red balls and one million black balls and you draw 1000 balls "at random" the 95-percentile sampling error is about 31. This means 95% of the time you draw between 468-532 red (or black) balls, even though the though the "expected" number is 500. Given that you drew 47% red balls (with a MOE of 3.1%) you would also report 532 black balls with the same MOE. The two events are not independent.

[alwaysconservative]

Wow! Bump and THANKS for all your hard work! I understand that you are evaluating the polls only on the terms of their own stated accuracy (MOE), and not their validity. I'm still not sure about the dynamics of a push poll (LATimes) on the actual result, for example. Some of the number shifts were due to external situations (Indian gaming money, the Arnold smear, and candidates dropping out), while others were probably due to forcing a vote in the survery, and not allowing an "undecided" response. And I would LOVE to see the internal polls from each party or candidate to see how they tracked with the public polls.

[R. Scott]

But, but … how will we know how to vote if the talking heads don’t tell us who is the favorite?

[OrioleFan]

Polling today is part of the political corruption in this country. Years ago, polls were used to find out what people think. Today, with the introduction of demographics, polls are used to influence the voting public. You can find a group of people (demographic) who will respond in a certain way to a certain wording of the questions. If the result is not exactly what you want, you alter the wording of question and run the poll again. This is repeated until you get the percentages to come out and that is the result that gets published. When you see a poll it's important to look at who commissioned it. You usually know what the results are going to be when ABC, CBS, Time, CNN, MSNBC, Newsweak, and any other liberal organization is paying the bills.

[Grampa Dave]

Thanks for your excellent work in this thread.

[BibChr]

bookmark

[Wolfstar]

Thanks, Onyx. The more informed we are, the better.

[JohnnyZ]

An honest poll works to suppress systematic error, there are so-called "push polls" designed to produce a certain result where every attempt is made to build in systemic error.

But what is a push poll anyway?

Fundamentally, what people label a push poll isn't a poll at all. A push poll is political telemarketing masquerading as a poll. No one is really collecting information. No one will analyze the data. You can tell a push poll because it is very short, even too short. (It has to be very short to reach tens of thousands of potential voters, one by one). It will not include any demographic questions. The "interviewer" will sometimes ask to speak to a specific voter by name. And, of course, a push poll will contain negative information - sometimes truthful, sometimes not - about the opponent.

http://www.cbsnews.com/stories/2000/02/14/politics/main160398.shtml

[Wolfstar]

A margin of error gives a pollster a plus or minus swing within which he can claim to be right. A typical MOE is +/- 3% of the percentage the pollster arrives at for a candidate or issue. Let's use an example of 45% +/- 3%. The pollster can claim to be right if the percent of the vote comes out at 42, 43, 44, 45, 46, 47, or 48 percent.

[BibChr]

Thanks for your work. I've wondered (mostly kiddingly) whether polling should be outlawed.

Dan
(c8

[Wolfstar]

Thanks for your reply. I appreciate it. First, I didn't pick the polls. They were the organizations doing polling on the recall. RealPolitics.com followed and listed them during the entire recall campaign. I simply took their list and tested the accuracy of the data the pollsters came up with against the actual election results.

Second, it is entirely random from several points of view, not the least of which is the unforeseen nature of the event, which set up a situation that was less influenced by media and other manipulation that may be typical in most elections.

Third, if polling is scientific, it should not matter if voter turnout is "unusual." Either pollsters can go out and accurately determine answers to their questions, or they can't.

You wrote: "Since polls are entirely dependant on the demographic choices made by the pollsters, Obviously, in an election such as the recall, where a large percentage of first time voters cast ballots, the odds of the polls being wrong are greatly increased." The whole point of my study is to challenge that very notion.

I challenge the notion that polls are at all scientific. Again, either pollsters can go out and accurately determine answers to their questions, or they can't. Why do they need ideal conditions? It's always the exception that proves the rule. So it seems to me that the unusual conditions of this recall were the perfect opportunity for pollsters to show their stuff. Yet they failed pretty convincingly.

Fourth, my premise, which is completely true, is that there is no objective way for the general public to verify the accuracy of most polls. This study is fully accurate within its own data. I made it a point to emphasize that what conclusions can be drawn are based on this study. Sure it would be nice to have the time and resources you describe, with hundreds of elections and polls to study and computer programs to sift through them, etc. But who in the general public has access to such resources?

The study shows what it shows, without artifice, and with a simple methodology that anyone can follow.

[Wolfstar]

You are most welcome. Thanks to Jim Robinson for giving us all the opportunity to have such dialogues.

[Wolfstar]

MOE's are given as PLUS or MINUS a percentage. So that I don't have to repeat myself, please be kind enough to see my posts #31 and #33 on this thread. Now, I don't claim to be a whizbang mathemetician. I try to reduce things down to a place where I can understand it, and assume that if I understand it, most others will, too.

If someone tells me they offer, say, a 45% prediction of something occuring, plus or minus 3%, that tells me they give themselves a range of seven percentage points within which they can claim to be right. Given an objective actual against which to compare the prediction, I can then see whether or not it was accurate. Very simple. Very clear. Very straightforward. The numbers speak for themselves.

[Wolfstar]

Hi, "AC." Thanks for adding your thoughts to this thread. Either pollsters can turn out an accurate product, or they can't. An unusual and unexpected event like the recall should have been an ideal opportunity for them to show their stuff, but they didn't. So how on earth are we in the general public supposed to know when they are accurate and when they are not? Your post buys into all the analysis of the situation the media fed us. That's fine. But what I'm doing is challenging the entire pollster/media edifice.

[Wolfstar]

Exactly!

[Wolfstar]

Polling today is part of the political corruption in this country.

Truer words were never spoken — er, written. :-)

People are free to accept or reject the results of this case study. But I would hope most would allow their assumptions about polling to be challenged a bit.

[Wolfstar]

Good morning, Grampa Dave. I appreciate your kindness. Just hoping to shake up a few cherished notions about polling.

[Wolfstar]

You are most welcome. Sometimes I wonder the same thing myself, LOL! Actually, what I've wondered for a long time are things like who invented "presidential approval ratings" and such, why, and how on earth can I trust something that I have no way to test for accuracy. The recall gave me such an opportunity. Most enlightening.