Skip to comments.Dialing Up a Controversy [Survey USA Polling Firm]
Posted on 08/01/2002 6:46:01 AM PDT by Coop
Survey USA, the leading firm in automated telephone polling, is the self-acknowledged black sheep of this family.
"I am not a member of their club," said SurveyUSA's JayLeve, who bills himself as the firm's editor. "I don't go out of my way to solicit their blessing."
Founded in 1992, Survey USA now boasts 50 local television stations as clients, according to Leve.
And yet, a number of established news entities -including The Associated Press, The Hotline and this newspaper - have a policy of not running automated dialing polls.
Leve clearly is irked by the disdain with which his company's methodology is viewed by the polling establishment and some news organizations.
"My issue is with the ridicule and intolerance and the censure that we have had to endure from people who not only have never looked at our work but are proud of the fact they have never looked,"he said.
At issue is the technique that SurveyUSA and other "auto dial" firms such as Rasmussen Research use to cull their information.
In a traditional survey, the polling firm develops and writes the questionnaire, which is then read to potential respondents by an employee of a phone bank.
Through a conversation with the polltaker, factors such as age, voting eligibility, ethnicity and other characteristics are determined and used to develop a sample representative of the electorate.
In a SurveyUSA poll, a taped message -typically utilizing the voice of a local news anchor from the station the company has a contract with -reads the questionnaire.Respondents punch certain keys to indicate their own demographic background as well as their preferences on a given topic or candidate.
Leve believes the lack of respect his firm has received began in 1995 with an article published inPublic Perspective Magazine by University of MichiganProfessor Michael Traugott.
In it, Traugott coined the acronym "C.R.A.P" -computerized response audience polls - to describe his belief that the methodology used bySurveyUSA was inherently flawed.
Leve noted that at the time of the article his firm had conducted more than 1,000 surveys in 75 elections and that Traugott refused to take an automated survey before penning his article.
"How can you rebut someone who won't take a minute to look at your work?" asked Leve.
Time has not changed Traugott's opinion about SurveyUSA and other firms like it, however.
"There is no sound theoretical basis for the way in which these surveys are conducted," said Traugott in an interview Tuesday.
His concerns are twofold. First, automated dialing polls have a much lower response rate than more traditional polls, he argues; second, it is very difficult (if not impossible) to screen out people not eligible or not planning to vote.
Traugott said the poll provides "a piece of data that might be of interest to the audience, but you don't really know who the respondents are."
Leve has a ready retort to Traugott's criticism. In an attempt to fool his own poll, Leve took a survey with the goal of not cooperating with the automated voice.
"The least amount of time it was technically possible to stay on the phone was two minutes and one second," said Leve, noting that in order to achieve such a short phone call he had to press 13 buttons at "precise intervals."
Leve believes no child or uninterested adult would stay on the phone for that amount of time solely to skew the poll.
"I can't tell you it's rocket science, but it's the most basic science," explained Leve. "We lead people through a series of win-win questions designed to start with adults and end up with likely voters."
As evidence of his polls' accuracy, Leve has compiled a chart comparing the final SurveyUSA poll in the 370 elections the company has been involved in since 1992 with the final polls done by competing organizations.
In the 105 contests in which SurveyUSA has matched numbers with Mason-Dixon Polling/Media Research Inc., the long-standing king of media polling, Leve claimed their data showed his company was more accurate 54 percent of the time, Mason-Dixon was more accurate in 33 percent of the races and the companies were "comparable" 12 percent of the time.
Of the 53 polling organizations Leve has researched, only 18 scored more accurately than SurveyUSA.
Leve said he commissioned the comparative study "because we were so tired of hearing from academics and the campaign managers of trailing politicians about how fundamentally flawed our methods are."
Brad Coker, president of Mason-Dixon, said that he would like to see the release dates and specifics of his firm's matchups with Survey USA before commenting, but did refer to Survey USA as "a competitor with a small 'c.'"
When asked about the validity of the automated dial practice,Coker said: "It is hard to believe that you are going to get a cross section of people who are registered, active voters to participate in these things."
Anecdotally,Coker cited an experience in his own home to disprove automated dial polls' effectiveness.
Coker, who lives in Baltimore, received a call several years back fromSurvey USA, which was at that time polling for WBAL, a local NBC affiliate. He said his 9-year-old daughter picked up the phone and participated in the survey.
ChuckTodd, editor of the Hotline, a daily political tipsheet, had a similar negative experience while taking a SurveyUSA poll.
"I was able to identify myself as a 19-year-old Republican Latina," said Todd. "Live callers could be deceived by respondents, but not like that."
The Hotline has not runSurveyUSA or any other auto-dial poll results since the 1998 cycle.
Jim Jordan, executive director of the Democratic SenatorialCampaign Committee, was more blunt.
"Survey USA numbers are the polling equivalent of a college professor's quote," he said. "That is, for lazy reporters they offer the ring of authenticity while in reality being cheap, uninformed, unreliable and meaningless."
Partisan pollsters are less willing to directly criticize Survey USA, choosing instead to focus on the differing aspects of the work they do.
"What they are doing is figuring out who is going to win, what we are doing is figuring out how to win," said Republican pollster GlenBolger, a partner in Public Opinion Strategies.
Fred Yang, a pollster with the Democratic Garin Hart Yang Research Group, agreed with Bolger's assessment.
"That's why political polls are much more rigorous methodologically and more expensive," he said. "The difference is we are in the business of winning elections and for that you need to have the most accurate look at the electorate, and the media is in it to make news."
I'd like to see an independent firm do the comparisons up above between SurveyUSA and the other polling firms.