Posted on 08/07/2006 7:49:16 PM PDT by shrinkermd
RATIONALE FOR REVIEW: On 17 July 2006 I opened the Star Tribune Newspaper. On the front page was a bold headline saying, “WIDE GAP A SURPRISE IN RACE FOR SENATE.”
The Star Tribune sponsored Minnesota Poll claimed that Democrat Amy Klobuchar held a strong, 19 point, lead over Republican Mark Kennedy. While the local Democrat Party poll also had her ahead by 15-16 percent, others found only a seven point difference (Zogby and Rasmussen).
While the poll did relate Democrat and Republican voting preference it did not give the proportion of Democrats and Republicans. This was surprising since in May of this year the same polling organization found 29% of the voters labeled themselves as Republicans and 25% labeled themselves as Democrats. This failure to clarify party membership made me uneasy since party preference usually predicts voting choice more than anything else. I did not assume the poll was fraudulent or deliberately skewed.
But in any case, something was wrong. Looking for an answer and an explanation lead me to search for recently published books on polling.
My search narrowed to a book by Professor George F. Bishop, “The Illusion of Public Opinion” (2004). I then tried to find a comprehensive review. I couldn’t. I bought it any way.
The book is well written, sourced and shows exemplary scholarship. Some readers might dislike literature citations, but the author uses these to examine the pros and cons of his conclusions. Who can question that?
As per the title the book does question some basic assumptions but the main thrust of the book is on methodology and how to do it better.
Professor Bishop believes polls have a place. His approach may be iconoclastic but he ends the book with suggestions on how to make political polling better.
OVERVIEW: This is a long review. For those in a hurry this “overview” may be sufficient.
The Title of this book begins with “The Illusion of Public Opinion.” The title disagrees with the common belief that “public opinion” is a concrete, measurable entity. There is no public opinion pool to be tapped.
Professor Bishop believes public opinion really occurs when someone questions respondents. Public opinion is a product of the questions asked. Properly targeted these questions may or may not lead to useful knowledge. Improperly targeted questions lead to mistaken conclusions.
Vague and misleading questions submitted to a public ignorant of the issues results in faulty opinion. To prove his point the author reviews a host of studies.
Before I summarize the conclusions of this book it is important to affirm the idea that public opinion is an illusion. This is easily done with a little reflection.
We have consciousness. Things enter and leave our conscious minds on a regular basis. Psychological cognoscenti call this the figure/ground formation and destruction. What is important is foreground and what is not important at the time is called background. In a healthy functioning person we do this regularly and automatically.
Foreground and background are most easily seen in basic needs. When we have not eaten for a long time we become hungry and this becomes foreground. When we eat, hunger perception disappears into the background.
Similarly, until we are questioned we seldom think about “public opinion” unless we are pundits, journalists or political aficionados. Even then we think only briefly and usually deliberately as part of a series of thought processes.
But once someone questions us for a poll that becomes the foreground and we opine on the basis of what we feel the answer is. To be always noted, is even the same question may elicit different foregrounds. For example a casual acquaintance asking, “How do you feel today?” has a different meaning than the same question asked by an examining physician.
Professor Bishop has taken relatively simple concepts and applied them to the methodology of public opinion surveys. These are the big problems of public opinion polling:
Politically attuned and well informed are a minority
Illusory opinions occur with vaguely worded questions
“As you may know” type questions lead to uninformed answers
Public opinion does not exist apart from the measurement process
Except for election polls there is no end point to measure accuracy
Respondients offer opinions on subjects they know little or nothing about
Identically worded questions may differ in meaning over time
Pollsters cannot really explain why people decide as they do
Polls are not substitutes for referendums
Professor Bishop relies on numerous citations to affirm his conclusions. I give some of the examples he uses to firm up and flesh out his conclusions.
QUALIFICATIONS OF THE AUTHOR: Professor Bishop is eminently qualified. He is a professor of political science at Michigan University. He directs a program credentialing pollsters. He has published numerous articles in a variety of prestigious academic journals. He is a senior research associate at the Public Policy Institute of Cincinnati.
CHAPTER I: THE ELUSIVENESS OF PUBLIC OPINION: I have already discussed the issue of what public opinion denotes.
What seems to be the worst case scenario for public polling is the inclination for polls to exert political control by creating the illusion that “the public has spoken in a definitive manner.” (Herbst 1993)
Filter and knowledge questions would seem to be mandatory for most polls but cost considerations make such efforts rare. Questions such as “who is running for ______ office” should occur before questioning about preference. Also, questions such as, “Do you feel sufficiently informed to give an opinion on________________.” The idea of filter and knowledge questions permits “don’t know” responses.
Cognitive psychologists generally agree that in survey questions it is essential that the question be so framed that it means the same thing to all respondents.
Presently, presidential popularity ratings are regarded as ongoing political referendums. Even small percentage moves are seen as important. This in spite of Crespi’s (1989) finding that the results varied according to whether asked in a dichotomous fashion such as “Do you approve or disapprove” or on a rating scale format such as “excellent, good, fair or poor.”
Converse (1987) and others have argued that journalists look at presidential approval polls as a “social telescope” when they really are a series of optical illusions.
CHAPTER II: ILLUSORY OPINIONS ON PUBLIC AFFAIRS: Here the author reviews the main points made about the illusion of public opinion.
He begins with a bang by citing the Gill study (1947) that asked questions about an imaginary “Metallic Metals Act.” Seventy percent offered an opinion.
Tom Smith took some Gallup data from the 1970s on the issue of balancing the federal budget. Ninety-six percent offered an opinion but 25% did not know whether the budget was balanced, 8% thought it was balanced and 40% knew it was unbalanced but not by how much.
From the series of literature cited, Bishop concludes that many answer to avoid the appearance of ignorance. Unless they are offered an opportunity to have a “don’t know” the answers should be suspect.
Recent studies in the Cincinnati area showed that 30-40% will offer an opinion on a nonexistent issue.
A social security survey came up with two sets of questions. One “individual” set asked, “Some people have suggested allowing individuals to invest portions of their social security taxes on their own, which might allow more money for their retirement but result in greater risk.” The majority (55%) answered in the affirmative. The second set of questions of an “impersonal” nature asked “Some people have suggested investing some Social Security funds in the stock market.” Contrary to the previous findings the majority (60%) thought this was a bad idea.
Malleability of public opinion seems to correlate with public policy ignorance. The issue then becomes the vagueness or ambiguity of the question rather than a measurement of public opinion.
CHAPTER III: SURVEY QUESTIONS AND REALITY: To begin with the form of the question is important.
Open ended and closed forms of questioning result in markedly different results. For example, Scherman asked a series of structured questions. He chose the four rarest answers of the question, “What is the most important problem facing the country today?” He then drew another sample and included this in a structured series of answers. The result was a full 60% of the new respondents chose one of the four previously rare answers as one of he most important facing the country today.
There is also a tendency to pick the “middle of the road answer.” When asked whether they are pro-choice or pro-life about 48% chose pro-choice and 42% chose pro-life. (Sadd 2003). But when the question was should abortion be legal under some circumstances, all circumstances and no circumstances, the results were: 55% some circumstances, 27% all circumstances and 16% no circumstances. The issue could, then, be reported either way based on whether you gave an opportunity for a middle position.
The Gallup organization in 2001 attempted to secure a firm opinion from the public as to whether the economy was getting better or worse. Only about 25% said it was getting better and 63% thought it was getting worse. About the same time a CBS poll was done asking, “Do you believe the economy is getting better, worse or staying the same?” Here, 44% thought the economy was the same, 43% worse and 10% getting better.
Another important issue is the wording of the question. Examples have been given before but a Chicago study in 1989 really highlighted the issue. Respondents were asked whether were spending on the poor was too little, too much or about right. The results were 63% thought we were spending too little. Then another series of samples were asked the same question but instead of poor “welfare” was used. Only 25% at most thought we should spend more!
The order of the questions is important. Ludwig in 1981 found that a majority (61%) would permit a married woman an abortion if she did not want any more children. This proportion dropped to 48% if it was mentioned the child was suffering from a birth defect.
Order of questioning includes such non-emotional items such as taxes. The Gallup organization in 1996 asked two series of questions. In the first instance the current system and flat tax were compared while in the second instance the flat tax question came first and the current tax question was second.
When the flat tax was presented last 52% favored the flat tax. When the flat tax was presented first only 43% favored the flat tax.
CHAPTER IV: THE CHANGING AMERICAN VOTER: FACT AND ARTIFACT: The author attempts to answer this difficult and contentious question. He concludes the differences over time are as much the result of differing questions as anything else. Even the same questions have different meanings over time. Some studies have found only a small percentage (9% or less) think in ideological or issue driven terms. A full 80% do not apply ideological tests to their choice of policies or candidates.
Besides being inconsistent at any point in time most voters are also inconsistent from one interview to the next. People vote more from emotion than ideology.
More often than not claims that public opinion has changed over time are illusory and dependent on test questions, the order questions are given and so forth. This particularly applies to questions regarding the loss of faith in government, loss of trust in government and loss of interest in civic duty.
CHAPTER V: SEPTEMBER 11’S EPHEMERAL OPINIONS: Shortly after 911 President Bush’s approval ratings hit an all time high. Similarly, Congress also had very high approval ratings and trust in government reached levels not seen since the 1960s. Even the influence of religion in American life looked stronger than ever. Everyone also agreed that “terrorism” was the number one agenda item.
This consensus was short lived. A year later these measurements reverted to baseline. According to the Gallup organization trust in government had drastically declined and, as prior to 911, over one half of respondents believed the “government was attempting too much.”
How to explain this ephemeral change? Professor Bishop believes the parsimonious explanation is the interpretation of the questions asked changed over time. The questions pre, immediately after and a year after were answered differently because the questions were interpreted differently.
CHAPTER VI: AMBIGUITIES OF MEASUREMENT: Pollsters love ambiguous questions. The classic example is to ask respondents to identify themselves as, “conservative, liberal or moderate.” What these terms mean to those polled is anyone’s guess. “Democrat, Republican or Independent” is similarly flawed with ambiguity. In spite of the ambiguity of terms, surveys now show an increasing tendency for “conservative” to be synonymous with “Republican” and “liberal” with “Democrat.”
The “trust in government” question has been asked for over fifty years. When all is said and done and all studies evaluated it appears the question correlates best with the public’s perception of the current President.
In spite of the Lewinsky investigation and its aftermath, President Clinton’s approval numbers not only remained high but they increased. This predicted the failure of impeachment effort. While variously explained these number, in my opinion, are easily understood if one assumes that President Clinton was a “celebrity.”
Celebrity status can be defined as people responding to a media figure “as if” they were real persons in their lives. President Clinton was seen as under attack for a personal foible. His ability to function in office is what was admired. His personal conduct was dismissed as a personal sin causing problems only for his family. The Republicans, and especially Newt Gingrich, failed to understand this and pursued an effort doomed to failure. I note that Professor Bishop also believes President Clinton’s approval numbers were a function of his functioning under stressful circumstances.
The author again concludes, by saying that much of what is called changes over time in public opinion is actually the result of ambiguous questions. The location of the problem is not in those polled but in the pollsters and their questions.
CHAPTER VII: SPURIOUS IMPRESSIONS IN THE PRESS: Professional pollsters are increasingly concerned about “pseudo-polls.” These are polls that give misleading conclusions based on loaded and unfair questions, selection bias and other techniques done to either make or shape news. The plethora of “pseudo-polls” creates the illusion that there is an informed public willing to opine on every conceivable issue.
Professor Bishop gives a many examples of mistaken polls. I will report on only two of them.
The first one concerns the “Contract for America.” Seen as a great coup for the Republicans and certainly worthy of emulation by the Democrats, the “contract” nonetheless has a fatal flaw. In 1994 an NBC/Wall Street Journal poll found that only 12% of Americans felt they were informed to a great or fair degree. About 9% had heard “some” about the contract and 79% had heard “nothing at all” or “not that much.” Polling by Los Angeles Times, Times Mirror and CNN replicated the study—all found the uninformed group to center around 75%. If people truly voted on the “Contract for America” they did so in spite of being uninformed.
The second one concerns “Holocaust Deniers.” A Roper survey of 1993 found that 34% said it was possible the Holocaust never happened. The problem with this survey was the question—“Does it seem possible or does it seem impossible to you that the extermination of the Jews never happened?” Of course, the question contained a double negative. Repeated studies found that when this was corrected for, only 5% or less of the population were Holocaust deniers.
CHAPTER VIII: ILLUSIONS OF CAUSALITY: ASKING WHY: Pollsters and the people paying them want more than poll results. They want to know what the results mean.
Usually what happens is poll results lead to conclusions as to what the results really mean. For example, they might state the President’s approval number is a reflection of public opinion on the war, tax policy, global warming and so forth. Sometimes they seek follow up questions from poll respondents to further buttress their beliefs.
These causal statements fail because they assume the respondents are making conscious decisions on policy matters in coming to their approval/disapproval decision. A false sense of causality results from this naive interpretation of results.
Respondents answer questions differently on the basis of the question’s order of presentation. Focusing attention on a limited number of possibilities restricts answers. The closed end question gives a spurious sense of informed decision making. When respondents are asked open ended questions they seldom even mention the closed end question alternatives.
The following is a Professor Bishop quote summarizing a host of findings.
“Theoretically speaking, the results presented here suggest that respondents, regardless of the form of the question asked—open or closed—give what they think is a plausible reason for their vote, rather than a causal reason. Indeed, respondents, as cognitive social psychologists would argue (Nisbet and Wilson), cannot really tell us about the causes of their behavior except perhaps under highly unusual conditions (of a gun to the head variety)”
Question form, response order and human nature are such that interpretation of exit and other polls as indicating this or that reason for a vote or a position on an issue have little or no merit. Survey respondents cannot explain their actions or their opinions; they can only offer after-the-fact rationalizations and plausible justifications.
CHAPTER IX: IMPROVING MEASUREMENT OF PUBLIC OPINON: Professor Bishop believes polls can be improved.
He goes back to George Gallup’s early work. Gallup urged questions on awareness and knowledge of issues, spontaneous overall views, specifics of the issue or proposal, reasons for holding opinion and intensity of feeling or belief.
Bishop also discusses the Howard Schuman’s “Random Probe Technique.” Here, closed-end questions are followed by random probing of the respondent as to the meaning of their answer.
Finally, the author reviews “The Mushiness Index” by psychologist and pollster Daniel Yankelovich. His idea is to determine the mushiness or firmness of the answer. The respondents are asked about their answers. Specific answers are examined for involvement, knowledge, engagement and degree of conviction.
Finally the author reviews pre-testing techniques and examining how closely the potential respondent is paying attention to the issue being polled.
CONCLUSIONS AND AFTERTHOUGHT: This review is long; however, it only summarizes a few of the many points of the book. The book is cheap, easily read and of extraordinary value.
Finally, in respect to the Minnesota Senate race I now feel that the poll was flawed. Among the flaws was demanding a voter decision too early in the election cycle. The poll was done in early July. The other/none and no opinion total number was only 13%. This low number suggests no filter question as to who is running, from which party and for what reason.
Perhaps the Zogby and Rasmussen polls were no better but a 19% difference is unlikely. The poll probably belonged deep within the paper as an interesting, unsubstantiated finding.
I am posting this late so as not to trouble the administrator with a length vanity.
fascinating, and should be widely read
Bump
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.