Polls with Large Voter Samples All Favor Romney; Smaller, Less Reliable Polls All Favor Obama. Why?

Pajamas Media ^ | 10/25/2012 | Zombie

[SeekAndFind]

RealClearPolitics publishes a continuously updated average of all major national presidential polls — as of this evening, their chart looks like this:

(Note that the RCP chart is updated frequently, so that by the time you read this, it may be slightly different.)

But whenever I check the RCP average, including today, I notice something odd: The larger the polling sample size, the more the poll favors Mitt Romney.

I’ve copied the RCP data and pasted it in here in a format that I can re-order. First, here’s how RCP organizes the chart, which they do chronologically, with the most recent poll at the top:

Poll	Date	Sample	MoE	Romney (R)	Obama (D)	Spread
RCP Average	10/15 – 10/23	–	–	47.8	47.2	Romney +0.6
Rasmussen Reports	10/21 – 10/23	1500 LV	3.0	50	46	Romney +4
ABC News/Wash Post	10/20 – 10/23	1394 LV	3.0	49	48	Romney +1
IBD/TIPP	10/18 – 10/23	938 LV	3.5	44	47	Obama +3
Gallup	10/17 – 10/23	2700 LV	2.0	50	47	Romney +3
Monmouth/SurveyUSA/Braun	10/18 – 10/21	1402 LV	2.6	48	45	Romney +3
CBS News	10/17 – 10/20	790 LV	4.0	46	48	Obama +2
NBC News/Wall St. Jrnl	10/17 – 10/20	816 LV	3.4	47	47	Tie
WashTimes/JZ Analytics*	10/18 – 10/20	800 LV	3.5	47	50	Obama +3
Politico/GWU/Battleground	10/15 – 10/18	1000 LV	3.1	49	47	Romney +2

But what if we simply re-ordered the polls not chronologically, but according to sample size, with the largest at the top? This is what it would look like:

Poll	Date	Sample	MoE	Romney (R)	Obama (D)	Spread
RCP Average	10/15 – 10/23	–	–	47.8	47.2	Romney +0.6
Gallup	10/17 – 10/23	2700 LV	2.0	50	47	Romney +3
Rasmussen Reports	10/21 – 10/23	1500 LV	3.0	50	46	Romney +4
Monmouth/SurveyUSA/Braun	10/18 – 10/21	1402 LV	2.6	48	45	Romney +3
ABC News/Wash Post	10/20 – 10/23	1394 LV	3.0	49	48	Romney +1
Politico/GWU/Battleground	10/15 – 10/18	1000 LV	3.1	49	47	Romney +2
IBD/TIPP	10/18 – 10/23	938 LV	3.5	44	47	Obama +3
NBC News/Wall St. Jrnl	10/17 – 10/20	816 LV	3.4	47	47	Tie
WashTimes/JZ Analytics*	10/18 – 10/20	800 LV	3.5	47	50	Obama +3
CBS News	10/17 – 10/20	790 LV	4.0	46	48	Obama +2

Notice the unmistakable trend?

All polls with 1000 or more respondents favor Romney; all polls with smaller than 1000 respondents favor Obama (or are tied).

Statisticians will tell you that the larger the sample size, the more reliable the poll. This fact is reflected in RCP’s “Margin of Error” (MoE) column, which shows a lower margin of error, and thus a greater level of reliability, for the large-sample pro-Romney polls. Each and every pro-Obama poll has a higher margin of error, and is thus less reliable.

These are the facts as they currently stand, and they’ve been true like this almost every day since soon after the first debate when Romney surged in popularity.

But Why?

The question we must now ask ourselves is why only the weak polls with low response rates favor Obama.

We can only speculate. I’ll posit some theories here, but I welcome speculations in the comments section as well.

• Theory #1: Non-respondents refuse to answer because they don’t want to reveal to a stranger that they’re voting for Romney.

As I noted in an earlier post, only 9% of people respond to poll questions: The other 91% either refuse to participate in the poll, or don’t even pick up the phone in the first place. I argued back then that it’s likely the majority of those non-responders are Romney voters, and this new polling trend confirms that thesis. It’s not clear why some pollsters have higher sample sizes than others: it could be that IBD/TIPP, NBC, WashTimes and CBS merely aimed low, and intended to have small poll samples — but it’s more likely that they tried to get more people to reply but had a significant non-response rate, and so “gave up” after reaching some number under 1000. Perhaps something about their methodologies, or the type of people hired to ask the questions, or their wording turned off or offended a certain type of voter inclined to favor Romney, and those people tended to hang up rather than state their preference for Romney. That would mean that IBD/TIPP, NBC, WashTimes and CBS are unintentionally under-sampling Romney voters, leading to their pro-Obama poll results.

• Theory #2: Certain pollsters are purposely publishing polls with high margins of error, so they can later have wiggle room when their pro-Obama slant proves to be inaccurate.

This theories supposes deviousness or malice on the part of certain pollsters: They want to sway the electorate with pro-Obama poll results, but want to give themselves room for plausible deniability if those polls later prove to be inaccurate. If challenged on their dubious published results, they can say, “Yes, our poll seemed to favor Obama, but you see his advantage was within the margin of error, so it’s plausible that it was just a statistical fluke, now that we see in retrospect that Romney won.” But a flaw in this theory is that two of the four pro-Obama polls were published by conservative-leaning media — IBD and Washington Times — who presumably have no interest in helping Obama.

• Theory 3: Sloppy polls simply give inaccurate results.

The smaller the sample, the more loosey-goosey the poll is, and the more likely it will produce an outlier result. But this fails to explain why all the outliers tilt toward Obama.

• Theory 4: The larger sample sizes also tend to be the more recent polls — all this trend reveals is that Romney’s lead is growing.

This is partly true, but there’s a lot of statistical “noise” in the recent/large/pro-Romney correlation. But it may account for some of the trend.

Otherwise, I’m stumped. I’ve love to hear what your theories are about this poll-sample trend. But perhaps it’s best to just bask in the knowledge that this week’s polls confirm that Romney has solidified his lead, and not worry about why some polls are off the mark.

[Wings-n-Wind]

Just follow the (walkin'around) money...

There ya go...

[Nervous Tick]

ping

[resistance]

1. Rats lie!
2. If the polls show R up by more than 3 a lot of dems won’t bother to vote.

[alancarp]

That’s a fabulous - and frankly pretty stunning - observation.

I’ll also point out that it’s the broadcast media tending to use the smaller sample sizes, which is a function both of their use of polls as a “news story” rather than for “science” (term used loosely), and of their desire to continually make their viewers feel that they have the freshest information available.

Rasmussen, for instance, uses moving averages taken over multiple days to stabilize the results. Clearly, the acronymn media doesn’t do this.

That’s my best guess... except for the obvious bias issues!

[Etpa]

I have a deep background in science/engineering and worst case analysis. What you present seems pretty clear to me to be an example of sampling bias in favor of Obama. The bias gets harder to sustain as the sampling size gets larger. My guess is that the actual overall average Romney numbers, without bias, would be a couple of points higher.

[carlo3b]

It’s less than two weeks to get the real numbers.. You should, for your own mental health, and practice a sound business judgement, don’t trust anything or anybody to not have an agenda in these polls.. My 2 cents/sense.. :)

[The_Reader_David]

I’m more interested in the response rate than the sample size. I think all the polls have become rubbish due to low response rate. I’ve been wondering whether Gallup, due to reputation, is getting a better response rate than the others.

[Cletus.D.Yokel]

Theory 5: Some conservatives (note: not labeled as “Romney voter” purposefully) are deliberately answering opposite their voting intentions to a) skew polls and give Obama some confidence and b) to further erode the credibility of the “Mainstream Media”.

The only poll I would answer truthfully would be Ras. If some media-bought pollster calls me, I’m 100% in Obamugabe’s camp!

[SDShack]

I have noticed this also. I think all your explanations have merit, but I rate #2 the most likely. The small lamestream focused polls are deliberately skewing the demographics to better than 2008 Democrat turnout. They know this is wrong, but it’s the only way they can keep their candidate in the lead. This is a deliberate attempt to drive the narrative that 0zer0 is ahead, or at least tied, when averaged with the polls that show 0zer0 behind. They need to drive this narrative to keep their base motivated for any Dem to have any chance in this election and try to avoid 2010 all over again. The high margin of error is their excuse for if it all goes wrong for them.
If 0zer0 is up 2-3 pts in their bogus poll, but there is a 3-4 point MOE, than that explains a Romney +1 to +2 win. Then add the “unexpected” Republican turnout, and you can “explain” the Romney +5 or +6 win. It’s all the lamestream media agenda to favor dems at all costs. They are nothing but an extension of the DNC. The lamestream media doesn’t care about their reputation anymore, or even the fact that they are biased. The big polling companies do, so that is why they are more accurate.

[deport]

The MoE dimishes in size as the poll samples get larger. The smaller the poll size the larger the potential for inaccuracies.

[muawiyah]

The sample size as reported may or may not have anything to do with how many responses they worked with. Given the PEW report on low response rates ~ 9% last year ~ we might have a situation where the response rates are actually as low as 3% this year.

That simply increases the influence of some smaller groups who always answer pollster calls (or take advantage of the call back opportunities most of them offer). That would mean that the smaller the response rate the more important homosexual responses would be, or those of the abortion industry, or maybe even professional feminists.

I think we need to know more about what 'sample size' really meant to the pollster. Plus, we all need larger grains of salt. Today's polls really aren't even a reliable gauge of trend line direction.

Final conclusion ~ this material definitely underscores what I've been saying about the effect low response rates have on polls ~ they become unrepresentative of the total population ~ and although the mathematics behind random sampling remains true, this is reality, and the closer the response rate approaches 0% the higher the probability of getting a BS answer.

[Hawthorn]

I've been computing weighted averages of the RCP-reported polls for about a week now. It's not the least bit difficult if you know how to do the most basic Excel operations.

Anyway, the current RCP unweighted average of ten polls has Romney up by 0.6%. But if we weight by sample by sample sizes, then Romney's advantage goes up to 1.2%

More interesting to me, however, is that when we drop the five polls that lack results from after the 3rd debate, then today's unweighted advantage for Romney goes up to 1.2%, and the weighted advantage goes up to 1.7%.

(I would also like to compute a pooled margin of error for the "meta-sample" -- except that my math skills don't seem up to the job!)

[Tennessean4Bush]

Statisticians will tell you that the larger the sample size, the more reliable the poll.

Not to put too fine a point on this but, you really should say that the larger the sample size the margin of error goes down. That may or may not mean more reliability. It does mean that given any other poll structural flaw, as you increase sample size your margin of error within the structure of the poll itself goes down. My main question with all these polls is this: if you get to the end of compiling your results and the internals don't pass the laugh test, you need to audit the structure of the poll. Does that mean you need to always adjust for party affiliation? No. But you should always watch that and make sure in the end what you get falls within the realm of possibility.

[FredZarguna]

I think the underlying correlation is really is that canvassing fewer voters = having less money = having poorer quality statisticians and actuaries.

There is another factor not considered, which is past history. In past elections, CBS/News, WSJ/NBC, and FoxNews/Opinion Dynamics always skew +2 to +4 Dem. Skewing in favor of those more likely to respond shows that these firms have poorer models to correct for the high refusal rates which are now standard.

[Nervous Tick]

>> I think the underlying correlation is really is that canvassing fewer voters = having less money = having poorer quality statisticians and actuaries.

==> poorer correction models (and less ability to test, verify, qualify and improve correction models).

Doing a good job of that modeling step can’t be cheap. Or easy.

Thanks and FRegards

[FredZarguna]

There is no meaningful "meta analysis" of these numbers, because the respondents in these polls are not cohorts in any meaningful sense. This is what's wrong with RCP's "poll averaging technique," and weighting doesn't really improve it, unless you are making the assumption that the smaller sigma poll is more accurate (reasonable if truly a random sample of truly likely voters, but those are two big IFS.) But anyway, if that's true, than why even include the large sigma poll to begin with?

The only real meta analysis you could do is to take the raw responses from these polls, apply your own analysis, and then weight the results. But you aren't going to get those numbers from these pollsters -- their corrections are already cooked in.

Poll averaging is junk math. It doesn't make the answer more likely to be right, it only makes the answer less likely to be wildly wrong. It's a form of hedging. But hedging is risk mitigation, it's not prediction.

[TomEwall]

There’s no reason a smaller sample size should be biased one way or the other. The chances are very small that all the small sample sized polls are leaning more towards Obama just by chance. More likely it has something to do with methodology. For example, are the polls favoring Obama weighted for a Dem turnout similar to 2008?

[Hawthorn]

>> There’s no reason a smaller sample size should be biased one way or the other <<

Agreed that there “should” be no difference in the bias between a large sample and a small sample. But over the past couple of weeks that I’ve been following the RCP averages, there does seem to be a remarkable pattern where the small-sample polls tend to favor Øbama and the large-sample polls tend to favor Romney.

My hypothesis to explain this pattern is that the small-sample polls are mainly low-budget undertakings where the poll takers and/or their financial sponsors have decided it would be too expensive to use “more representative” polling techniques, meaning that they tend toward having samples that over-represent Ø-leaning LV’s.

[Hawthorn]

>> There is no meaningful "meta analysis" of these numbers, because the respondents in these polls are not cohorts in any meaningful sense. This is what's wrong with RCP's "poll averaging technique," and weighting doesn't really improve it, unless you are making the assumption that the smaller sigma poll is more accurate (reasonable if truly a random sample of truly likely voters, but those are two big IFS.) But anyway, if that's true, than why even include the large sigma poll to begin with? <<

Why include indeed! Whether you like the situation or not, and whether I like it or not, the RCP average is a "fact of life" for the pundit class. It's not going away.

Given this reality, and given my belief that sometimes you gotta go into battle with the "army you have" rather than the army you "wish you had," I think its overwhelmingly likely that an average weighted by sample size has a higher probability of being closer to the true, underlying number than does an unweighted average.

If you disagree, that's fine by me. But if somebody is willing to place a bet as to which average will be closer to the final outcome, I'll put my money on the weighted version.