Free Republic Browse · Search News/Activism Topics · Post Article

Survey Methodology [Zogby America Likely Voters] 12/12/07 thru 12/14/07 (Math ExpertsPlease explain)
Zogby.com ^ | December 19, 2007 | Zogby International

Posted on 12/22/2007 3:52:38 PM PST by Yaelle

Survey Methodology [Zogby America Likely Voters] 12/12/07 thru 12/14/07

This is a telephone survey of [likely voters] conducted by Zogby International.

The target sample is [1000 likely voter] interviews with approximately [96] questions asked. Samples are randomly drawn from telephone cd’s of national listed sample. Zogby International surveys employ sampling strategies in which selection probabilities are proportional to population size within area codes and exchanges. Up to six calls are made to reach a sampled phone number. Cooperation rates are calculated using one of AAPOR’s approved methodologies[1] and are comparable to other professional public-opinion surveys conducted using similar sampling strategies.[2] Weighting by [region, party, age, race, religion, gender] is used to adjust for non-response. The margin of error is +/- 3.2 percentage points. Margins of error are higher in sub-groups.

The target sample is [436 Democratic Likely Primary Voter] interviews with approximately [96] questions asked. Samples are randomly drawn from telephone cd’s of national listed sample. Zogby International surveys employ sampling strategies in which selection probabilities are proportional to population size within area codes and exchanges. Up to six calls are made to reach a sampled phone number. Cooperation rates are calculated using one of AAPOR’s approved methodologies and are comparable to other professional public-opinion surveys conducted using similar sampling strategies. Weighting by [party, age, race, religion, gender] is used to adjust for non-response. The margin of error is +/- 4.8 percentage points. Margins of error are higher in sub-groups.

The target sample is [432Republican Likely Primary Voter] interviews with approximately [96] questions asked. Samples are randomly drawn from telephone cd’s of national listed sample. Zogby International surveys employ sampling strategies in which selection probabilities are proportional to population size within area codes and exchanges. Up to six calls are made to reach a sampled phone number. Cooperation rates are calculated using one of AAPOR’s approved methodologies and are comparable to other professional public-opinion surveys conducted using similar sampling strategies. Weighting by [region, party, age, race, religion, gender] is used to adjust for non-response. The margin of error is +/- 4.8 percentage points. Margins of error are higher in sub-groups.

[1] See COOP4 (p.38) in Standard Definitions: Final Dispositions of Case Codes and Outcome Rates of Surveys. The American Association for Public Opinion Research, (2000).

[2] Cooperation Tracking Study: April 2003 Update, Jane M. Sheppard and Shelly Haas. The Council for Marketing & Opinion Research (CMOR). Cincinnati, Ohio (2003).

(12/19/2007)

KEYWORDS: methodology; polling; zogby
Each time I read a poll, I wonder how the information is gathered. The first thing I think about is caller ID. Many people with full lives will check the ID and not take calls except from people they know. Many people deliberately avoid polls, and many people are simply too busy, even if they took the call, to continue with it. It sounds like the Zogby calls involved tens of questions (the site as quoted above says 96, which I am finding hard to believe).

Is there some kind of automatic screening going on, simply by who chooses to answer the phone and continues with a very long polling session? And how does that correspond to Actual American Voters, some of whom are busy and don't answer polls?

Also, the sampling strategy for lower population sizes: does this really work? Does it end up meaning that if only two Asian men were sampled, it would be extrapolated to signify some absurd percentage of Asian men prefer one candidate to the other?

I know there is some science behind statistics and surveys, and pollsters like to show how close they were AFTER an election, but I keep coming back to little things like Caller ID. Is anyone explaining that the samples are heavily weighed toward either people who do not yet have caller ID or toward people who love speaking with pollsters?

1 posted on 12/22/2007 3:52:40 PM PST by Yaelle

To: Nevadan; Greysard; bs9021; bboop; RightField

Ping to explain polling methodology!

2 posted on 12/22/2007 3:56:26 PM PST by Yaelle (FRED, the most intelligent choice)

To: Yaelle
The methodology is simple. Build up the Republican that is most likely to lose if nominated for as long as possible. In the last few days before ballots are actually cast try to bring a little reality into the polls so that our reputation as pollsters can be salvaged.
3 posted on 12/22/2007 3:57:23 PM PST by hflynn ( Soros would not make any sense even if he spelled his name backwards)

To: RightWhale; Clint N. Suhks; Lil'freeper; The_Reader_David; 1rudeboy; Onelifetogive

Ping to help explain why we should believe polls, mathematically, please?

4 posted on 12/22/2007 3:59:45 PM PST by Yaelle (FRED, the most intelligent choice)

To: hflynn

Are you saying that polls are flawed from the start due to the “push-polling” that goes on with the questions pushing the poll “victim” into one direction?

We’ve all had polls like that, in which the “neutral party” on the line is asking questions like “If you knew that Joe Opponent burnt kittens in his fireplace, would you still be likely to vote for him?”

However, I am assuming that Zogby’s questions are more neutral and not “push polling.” Am I right?

5 posted on 12/22/2007 4:02:23 PM PST by Yaelle (FRED, the most intelligent choice)

To: Yaelle
Zogby polls proven primary voters.

Everybody has an opinion, and lots vote in actual elections, but few vote in primaries.

Zogby targets actual past primary voters.

To: r9etb; napscoordinator; ChiefChris

If we really “live and die” by the poll numbers around here, I’d like to know the basis of the science of calling and bugging people, and how it relates to real voters actually voting!

7 posted on 12/22/2007 4:06:01 PM PST by Yaelle (FRED, the most intelligent choice)

So the ones he includes are very likely voters on the basis that they DID vote in the last primary. OK, good. But how many of them decide not to go ahead with the entire poll because they are busy? Is his poll top heavy with people who have a lot of time on their hands?

8 posted on 12/22/2007 4:07:50 PM PST by Yaelle (FRED, the most intelligent choice)

To: Yaelle

I think they gather the samples and then they get demographics and apply a statistic test which include error

error = sqrt ((p X (1-p))/(n-1))

p (candidate A) = 0.48
q (candidate B) = 0.52
n = 502

error = (sqrt(0.52 X 0.48)/(501)) =0.02

Candidate A = 48%
Candidate B = 52%
Margin of error = +/- 2%

If n > 33, then n-1 is not considered statistically significant.

9 posted on 12/22/2007 4:09:12 PM PST by Perdogg (Fred Thompson for President)

To: Yaelle
Zogby has an agenda that is pro-Arab. Zogby’s brother is James “Jim” Zogby, founder of the influential Arab American Institute which is not what I would call unbiased. Jim Zogby is also employed part-time by Zogby polling. You figure it out but Zogby polls are somewhat ourregeous until the votes are about to be cast. All of a sudden huge Democrat margins evaporate.
10 posted on 12/22/2007 4:16:32 PM PST by hflynn ( Soros would not make any sense even if he spelled his name backwards)

To: Perdogg

I love your math but I am just not completely understanding it. What is the variable for the “demographics” and how does that work in? How is significance determined?

Please pretend you are speaking to an absolute idiot (me) with blonde hair.

11 posted on 12/22/2007 4:22:32 PM PST by Yaelle (FRED, the most intelligent choice)

To: Yaelle
I personally find the very notion of polling to be repugnant.

In my view only an idiot would respond to a poll of any kind. I never answer local census forms when they arrive, but the national census is something that I feel bound to respond to even though I resent it. If I ever get one of those comprehensive national census forms I’ll burn it.

The problem for me is that somebody is responding to pollsters somewhere, everyday, and in large numbers. In the case of Zogby polls, they are usually better than most.

I do believe that Zogby is biased and has been doing some tweaking during the last two presidential elections.

They were wrong both of those years.

In the lesser publicized elections, the Zogby polls are quite good, and the hacks use them....a lot.

To: Yaelle

Zogby, unlike say Rasmussen, just makes it up.

13 posted on 12/22/2007 4:26:34 PM PST by Checkers (First they came for the Mormons, but I said nothing because I was not Mormon.)

To: hflynn

I have heard that his brother is the head of the American Arab Institute. I do not know if that makes Zogby biased. His polls seem to be very trusted on both sides; why, if his family is so un-neutral?

How would bias be hidden by a poll? How to bias your poll and yet after the election be the one who had the closest results? AND WHY ARE POLLS TAKEN SO SERIOUSLY, by basically everyone (FReepers included)?

Is there really only a tiny MOE (margin of error)?

Do we talk poll numbers all day and all night just to have SOMETHING to talk about?

Why should we believe polls?

14 posted on 12/22/2007 4:28:09 PM PST by Yaelle (FRED, the most intelligent choice)

To: Yaelle

The demographics would be separated out into sub categories and the same statistically test applied.

If you live in a district where one party is significantly more represented on the voter roles, then the number of that party may be sampled more than other parties on a pro rata bases.

Eg, if I have a district in Utah that is 75% republican, I may want to have a sample size of 75% republicans. You may perform another survey, lets say 55% republicans and then compare the samples.

This of course, is not cheap.

15 posted on 12/22/2007 4:29:04 PM PST by Perdogg (Fred Thompson for President)

To: Checkers

OK, then please explain Rasmussen. Explain the methodology of ANY telephone poll you actually TRUST. And why should we put faith in it?

I mean, if your father is in surgery and the doc comes out to give you an update midway through, of course you hang onto every raised eyebrow or sigh in his delivery. Is that what we are doing when the polls come out?

16 posted on 12/22/2007 4:30:33 PM PST by Yaelle (FRED, the most intelligent choice)

To: Perdogg
Eg, if I have a district in Utah that is 75% republican, I may want to have a sample size of 75% republicans.

This makes sense to me only if it's a general election.

17 posted on 12/22/2007 4:32:29 PM PST by Yaelle (FRED, the most intelligent choice)

So Zogby tweaks and can’t be trusted in big elections, but is thought to be trustable in the smaller races?

I too think that many people would either NOT pick up the phone for a pollster (unknown caller ID) or even refuse to go further if they’d got him on the line. Isn’t that a screening right there? How do we trust polls, when the headline never says “ROMNEY IS WINNING 2 to 1 OF LIKELY VOTERS WHO CHOOSE TO TALK TO POLLSTERS”?

18 posted on 12/22/2007 4:35:21 PM PST by Yaelle (FRED, the most intelligent choice)

To: Yaelle
If I am telling you the polls are bias I am also telling you I don't take them seriously. Zogby's polls aren't trusted because of the wide variance they have leading up to the actual casting of ballots. Zogby relies on his last poll to be the 1 his credibility is based on. If he takes a 101 polls and the 1st 100 are wrong we are all expected to remember the last 1.

Zogby predicts Kerry electoral landslide! by daninoah Tue Nov 02, 2004 at 02:23:55 PM PST He says that Kerry will will 311-213, but barely lose the popular vote. Kerry wins Ohio, Florida, and most of the other battlegrounds.

19 posted on 12/22/2007 5:16:14 PM PST by hflynn ( Soros would not make any sense even if he spelled his name backwards)

To: Yaelle
Many people with full lives will check the ID and not take calls except from people they know.

Even without talking to people with Caller ID one can impute what answers they would give.

20 posted on 12/22/2007 5:24:23 PM PST by trumandogz (Hunter Thompson 2008)

To: hflynn
Zogby predicts Kerry electoral landslide! by daninoah Tue Nov 02, 2004 at 02:23:55 PM PST He says that Kerry will will 311-213, but barely lose the popular vote. Kerry wins Ohio, Florida, and most of the other battlegrounds.

Thanks for this little reminder of Zogby's/pollsters' infallibility!

21 posted on 12/22/2007 5:26:44 PM PST by Yaelle (FRED, the most intelligent choice)

To: Yaelle

You have it backwards. An election that does not match a predetermined poll result is a rigged election. An electorate that goes against a poll result is too stupid to see that they have been deceived.

22 posted on 12/22/2007 5:42:15 PM PST by depressed in 06 (Bolshecrat, the amoral party of what if and whine.)

To: Yaelle
Lets ping to ask why we have polls????

The only thing that can be “gained” by one is to influence a vote!!!

Who needs to know how I’m gonna vote??...read everyones vote in the paper the day after the election...

23 posted on 12/22/2007 6:00:38 PM PST by M-cubed (Why is "Greshams Law" a law?)

To: Yaelle

Remember that Zogby is a Muslim activist and anti-American. His polling is always suspect.

24 posted on 12/22/2007 6:43:46 PM PST by Mobile Vulgus

To: hflynn
Zogby has an agenda that is pro-Arab.

I don't think there's any evidence to show that that's true at all.

Zogby (the pollster, not the brother) is a Democrat but he's nonetheless a good pollster.

A lot of people don't answer the phone anymore, but a good random sample will take that into account, set a target for the number of respondents it needs to establish statistical probability and the pollster will just keep dialing until that number is reached.

The big problem nowadays is that so many people no longer have land-lines. Younger people are excluded in larger percentages but increasing numbers of older potential respondents are also not included. Nonetheless, the polls are still statistically valid (IMO) for the most part.

You have to pay attention to how the questions are phrased and keep in mind that they represent only the proverbial "snapshot" in time. Take them only for what they're worth, especially at this point. The match-ups between Hillary and, say, McCain are essentially worthless except for entertainment value.

But with each day that passes, the Iowa polls become more valuable as an indicator of how Iowans are thinking.

25 posted on 12/22/2007 6:54:26 PM PST by BfloGuy (It is not from the benevolence of the butcher, the brewer, or the baker, that we can expect . . .)

To: Yaelle

I am one of the folks Zogby polls in their online polls. They seem to accept my answers for the most part, but several times I have been in the midst of a poll and the answer I gave was NOT what they were wanting or expecting. I get shuffled aside into a generic questionnaire, and then I don’t get any polls for 3 weeks.

Zogby is biased, they just hide it well.

26 posted on 12/22/2007 11:58:25 PM PST by Don W ( Police were called to a day care where a three-year-old was resisting a rest.)

To: Yaelle

zombie is a democrat. ‘nough said?

27 posted on 12/23/2007 4:18:40 AM PST by jmaroneps37 (Conservatives live in the truth. Liberals live in lies.)

To: BfloGuy
I don't think there's any evidence to show that that's true at all.

Zogby (the pollster, not the brother) is a Democrat but he's nonetheless a good pollster.

James Zogby is Founder and President of the Arab American Institute and Senior Political Consultant to Zogby International.

In April of 2002, over 5 years ago, Christian Science Monitor hosted a breakfast with John Zogby (the pollster) and James Zogby (president of the Arab American Institute). The Zogbys described the Zogby poll as being run by BOTH of them, not just by the pollster. Many appearances on TV to discuss Zogby Polls involves both of them. The formula is John talks about the poll itself then laterals to Jim who attacks the anti-Arab U.S. foreign policy.

The Zogby Poll brothers are joined at the hip and the pro-Arab bias is real.

28 posted on 12/23/2007 4:57:34 AM PST by hflynn ( Soros would not make any sense even if he spelled his name backwards)

To: Yaelle
If we really “live and die” by the poll numbers around here, I’d like to know the basis of the science of calling and bugging people, and how it relates to real voters actually voting!

In theory, if you can select a truly random and representative sample of the voting public, then a poll can provide a very powerful tool to measure the probable voting results, even from a relatively small sample.

You can see how that would be attractive to a candidate -- both as a way of shaping his message, and of deciding whether his millions of dollars of campaign money are a waste of time. And of course it's attractive to media organizations, and to us, because it's fun to see "who's winning."

The problem is that it's becoming more difficult to actually collect a truly random, representative sample, because of cell phones (which they can't or don't call), no-call lists, caller ID, and plain old "I don't want to talk to a pollster" attitudes, and so on.

Plus which, they want to capture the whole country in a few hundred calls, so the odds of sampling bias are high: Zogby has has famous "special sauce" for adjusting his results to account for the "real" proportions of Republicans, Democrats, etc, as opposed to the proportion in his sample.

On the largest scale, I think polls are still reasonably accurate for distinguishing between primary candidates -- you only poll likely primary voters, and so the uncertainty of independent voters tends to go away. But then, you'd pretty much know that without polling: Tom Tancredo shouldn't have needed a poll to know that he was never going to get out of low single-digits.

Fred Thompson is an interesting case: I think he's "outside" the polling right now -- polls are not reaching the voters who are likely to vote for him in the early primaries. He may be artificially low. We'll see if I'm right about that ... it's just the sense I get from watching how folks respond to him.

29 posted on 12/23/2007 6:08:57 AM PST by r9etb

To: Mobile Vulgus; Yaelle
Remember that Zogby is a Muslim activist . . .

He is?

According to this page he is the son of Lebanese Catholic parents. There is no mention of a conversion to Islam.

I know I am arguing from silence but you'd think there'd be a mention in the article if he went from being a Catholic to being a Muslim.

Arab != Muslim.

30 posted on 12/23/2007 6:21:30 AM PST by ksen ("For an omniscient and omnipotent God, there are no Plan B's" - Frumanchu)

To: r9etb
Yours is the post I really wanted to see. I think you have finally explained it to my satisfaction. Articles describing poll results always leave out the real facts, which are, like you say,

The problem is that it's becoming more difficult to actually collect a truly random, representative sample, because of cell phones (which they can't or don't call), no-call lists, caller ID, and plain old "I don't want to talk to a pollster" attitudes, and so on.

I can't think of a way to TRULY collect a random, representative sample. I do know that the faster the call would be would help. For instance, if the pollster got someone and was quickly able to say that they are doing an independent poll for the presidential election (something a FReeper would never believe, anyway) and the call would last a total of 2 minutes, you might get a larger sample. Also if they were allowed to call cell phones. The pollster would list the candidates in the pollee's party, and ask them to pick. Maybe get some demographics really fast. Boom, the pollee's life would not be too disturbed.

Otherwise they really are only sampling people without caller ID who have nothing better to do.

Fascinating to me that you think Thompson voters are outside the polling population currently. I asked this whole thread because I was getting that impression but wasn't quite sure. It's just too hard to know.

31 posted on 12/23/2007 12:23:43 PM PST by Yaelle (FRED, the most intelligent choice)

To: hflynn
The Zogby Poll brothers are joined at the hip and the pro-Arab bias is real.

Whoa. Interesting.

Now I wonder about the other pollsters. I don't wonder too much about CNN, though...

32 posted on 12/23/2007 12:25:13 PM PST by Yaelle (FRED, the most intelligent choice)

To: Yaelle

It would be easier to say “mix one cup mayonnaise in a bowl with half a cup of dill relish (drained), a tablespoon of worcestershire sauce, three tablespoons of lemon juice with pulp, and half a teaspoon of rosemary. Whip thoroughly and chill for two hours. Use within three days.”

33 posted on 12/23/2007 12:27:55 PM PST by Petronski (Who has 47% core opposition? Hillary . . . . and Slick Willard!)

To: Yaelle
Sampling

1.) cell phones may be excluding the youth vote, but that demographic often has the lowest voting record (by age). Therefore, any adjustments by age are going to maximize the elderly vote, which is historically overrepresented, and minimize the votes of the poor or the young, who typically vote in much lower numbers.

2.) At least Zogby is using a 1,000 person sample, which is generally considered the optimum number of respondents in a survey. However, since Zogby is breaking it into 2 groups (Republicans and Democrats), and then by demographic (ethnicity, socioeconomic factors, voting record, geography, etc.), the 1,000 count increasingly becomes meaningless. It's better to look at both whether the poll by demographic is statistically relevant and the margin of error, because organizations may choose to publish results which has no meaning whatsoever, but simply look good.

Push Polls v. Neutral Polls

Not only have I worked on several campaigns, but I also visited inside a polling business. Several things:

A.) Polling companies typically need to affiliate with a particular party in order for local representatives and companies to trust that their information will be kept confidential. It's common sense-- if you're a rabid Republican, why would I let you conduct the in-depth survey or poll in Maryland about Kerry's popularity? Quality polling businesses hire interviewers who have little political exposure and no accent, so that they have little or no opinion about the results and people can actually understand them.

B.) If your questionmaire terminates or turns generic, it is likely that you indicated that you are in a demographic for which they already have sufficient respondents. In order to avoid overrepresenting say, pro-life Republicans, in a poll, they will either end or divert the questionnaire in order to maintain their original demographic requirements.

C.) Each candidate will broadcast the results of whichever poll makes them look best, and, conversely, selectively underplay the sections of a poll which makes him/her look bad. The same thing goes with the news. Therefore, if a candidate is being touted as doing extremely well in the news, it could be the channel's fault more than the polling organization.

Zogby & Rasmussen vs. The Rest

What really gets me is when news stations such as ABC arrogantly broadcast a "new poll" which they themselves designed--- often with less than 500 respondents, non-random, using the snowball method. Then they tout this great new poll as "America Speaking" when it really means nothing. Be thankful that polls such as Zogby & Rasmussen--which at least attempt for neutrality--exist. Who'd prefer a CNN poll every night?

34 posted on 12/23/2007 6:46:19 PM PST by bs9021 (facts speak loudly)

To: bs9021

Good points and it makes sense. The only polls I have ever taken part in were for local candidates and were very obvious push-polling, asking stupid questions that did not allow my “real” opinions. I was forced to choose between “being stabbed or shot to death,” if you know what I mean. BS like “If you had to choose which issue was the most important to you, would you choose global warming or helping the homeless?”

I’d like to hope that the Big ‘Uns like Zogby and Rasmussen actually stay very neutral in their polling even though many have said they do have their biases. The demographic representation thing still spooks me a little.

But as another poster already said, it’s fun to have colorful charts and numbers to work with. And if we KNEW that the results were significant, at least we’d get some knowledge that Candidate A “moved up two points” from the previous poll, etc.

35 posted on 12/23/2007 11:35:29 PM PST by Yaelle (FRED, the most intelligent choice)

To: Petronski

You forgot to add “Place over television talking head pollster’s head, then cover with a garbage can” (a la Carville).

36 posted on 12/23/2007 11:36:45 PM PST by Yaelle (FRED, the most intelligent choice)

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

 Free Republic Browse · Search News/Activism Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794