Mark DiCamillo of the Field Poll, and Mark Baldassare of the Public Policy Institute of California, discussed polling in the 2010 election at the Sacramento Press Club this week. DiCamillo talked about the rising influence — and the problems with — robopolls like Rasmussen, Pulse Opinion Research and others. What he had to say is too important to be heard only by the hacks of the press corps. Here’s an edited version of DiCamillo’s remarks for the Calbuzz cognoscenti.
By Mark DiCamillo
Director of the Field Poll
When comparing the polling in California this year to previous years, two things stand out. The first is the sheer number of pre-election polls conducted and reported. When reviewing only polls conducted in California’s general election races for governor and U.S. Senate, by my count there were at least 75 different statewide public polls completed by 14 different polling organizations. And this doesn’t even count the many private polls conducted in each contest for the various political candidates and campaigns.
The other thing that jumps out is that in this election cycle more of the polls than any year previously were robopolls, also referred to as Interactive Voice Response or IVR polls.
By my count nearly half of all of the statewide public polls reported in California’s general election in the governor and Senate races were robopolls. If you were to also include polls conducted in local election contests across the state, robopolls constituted the majority of all public polls in California in this year’s general election.
Because robopolls are now so prevalent, it is more important than ever for the media and the public to understand just how these polls differ from traditional telephone polls, especially those conducted by the state’s three leading public polls – The Field Poll, the Public Policy Institute of California and the Los Angeles Times/USC Poll.
Comparing the methodologies: The basic survey approach of robopolls is to contact people by telephone using the recorded voice of a professional announcer. The announcer instructs those answering the phone to use the keypad on their phones to answer their poll questions. Traditional polls use live interviewers to call voters who ask each question directly.
Because most of the costs of conducting a traditional telephone survey are derived from the time spent and wages paid to telephone interviewers and their supervisors when carrying out data collection, robopolls are much cheaper to conduct than a traditional telephone survey since there are essentially no interviewer costs associated with conducting these polls.
This is one of the reasons why they are now so prolific. They can be conducted at a fraction of the costs of conducting a traditional phone survey.
However, other than cost, robopolls differ from traditional telephone surveys in a number of important ways.
(1) Short polling period, no callbacks
Most robopolls are typically conducted very quickly over a one-day period. They typically make only one attempt to reach a voter at each number dialed. If no one answers the phone they do not make callbacks to that number but simply replace it with a new telephone listing.
By definition, this means that robopolls have significantly lower response rates than traditional polls and are polling only those segments of the voting public that are the easiest to reach.
Contrast this to a Field, PPIC or Times/USC poll which is typically conducted over a one-week period and which makes up to six to eight different attempts at each usable number to try to bring voters into their samples. While this is more costly and time consuming, it produces samples that more closely capture the varied demography of California’s voters — working and non-working, old and young, white non-Hispanic and ethnic, those living alone and those living in multi-family households.
(2) Limited knowledge about who is actually answering their questions
Robopolls make calls from a random digit dial sample of all possible residential landline telephone numbers within the political jurisdiction they are polling. The recorded announcer instructs the person answering to tell them if they a registered voter, leaving this important selection criteria totally in the hands of the respondent.
Polls like The Field Poll, the Times/USC Poll and virtually every private poll conducted for a political campaign sample voters off of lists derived from the state’s official voter registration rolls.
This gives the poll a number of advantages. First, it enables interviewers to ask to speak to a specific individual by name and if that individual is not available, the interviewer can make appointments to call back that voter at a later time. Also, because the sample of names is derived from lists of known voters, we know by definition that the person we are seeking is indeed a registered voter. Working off a voter list also provides the pollster with the voter’s actual party registration as well as their frequency of voting in past elections, since this information is contained on the official voting records. This information can also be used to ensure that the sample is aligned properly to the state’s actual party registration and in identifying which voters are most likely to vote.
(3) Exclusion of cell phones
By law, the automated dialing devices used by the robopolls are not allowed to call cell phones. Traditional telephone polls routinely dial cell phones by hand to include them into their samples. Since more than 20% of all California voters are now cell-phone-only households and cannot be reached when dialing random sample of landline phone listings, most robopollsters are systematically excluding these voters from their samples.
(4) Language limitations
To my knowledge, the pre-recorded messages of most robopolls are in English only. This excludes from their samples the additional set of voters who do not understand spoken English. By contrast, Field, PPIC and the Times/USC polls routinely conduct all of their statewide polls in English and Spanish. In addition, Field’s final pre-election poll this year was extended further to include four other Asian languages and dialects — Cantonese, Mandarin, Korean and Vietnamese.
We estimate that 7%-10% of all registered voters in California would either prefer or require non-English language interviewing when completing a telephone survey, so this portion of the state’s fast-growing ethnic voters is under-represented by the robopolls.
(5) The need to construct a model and apply larger weighting adjustments
Each of these factors means that the quality of the raw unadjusted survey data derived from robopolls is of significantly lower quality than that of traditional telephone polls like those conducted by Field, PPIC and the Times/USC.
In their methodological descriptions, robopollsters admit that women are much more likely to participate in their surveys than men, and that older voters are included in their samples in far greater numbers than young or middle age voters. Because their initial data are less representative, robopolls need to make fairly major adjustments to their raw data to bring their samples into balance with the characteristics of the larger voting population.
By contrast, the unadjusted raw data obtained by traditional telephone pollsters more closely reflect the actual population of voters they are polling. While The Field Poll does make weighting adjustments to its samples, the adjustments tend to be small and have a modest impact on the overall poll’s statewide findings.
For example, The Field Poll’s final pre-election poll this year showed both Democrats Jerry Brown and Barbara Boxer ahead of their Republican opponents in this year’s races for governor and U.S. Senate by eight to 10 percentage points in both our unweighted and weighted samples. The main impact that the weighting or sample adjustments was to align the various subgroups to known characteristics of the voter population. Importantly, they did not have much impact or significantly alter the overall statewide preference distributions initially found in the survey.
Despite their sampling drawbacks, the better robopollsters are able to transform the survey information they obtain into reasonable pre-election poll estimates by developing a sophisticated model of the probable electorate and adjusting their sample to conform to its characteristics. Because of this, I view the better robopollsters more as skilled modelers of the electorate than as high quality survey researchers.
But because of their need to construct models to determine the overall shape of the probable electorate rather than rely on actual survey data or information about each respondent’s voting record to determine this, the modeling itself can create potential problems.
For example, the determination of how many Democrats and Republicans to include in a sample is closely tied to voting preferences. When making this determination the robopollster takes great liberties in deciding who is ahead and by how much, since even a slight change in the partisan distribution of the sample will affect the preference distributions in most election contests.
This is perhaps the most worrisome aspect of the robopoll method, since it confers robopollsters with greater latitude in influencing the outcomes of their poll measures, and risks either introducing systematic biases into their poll.
Some robopollsters admit to taking into account a state’s voting history, national trends and recent polling to construct their partisan weighting targets. This means that the proportions of Democrats and Republicans allocated to their sample are derived from subjective judgments about the historical and prevailing political conditions in a given state and from other polls already conducted in that political jurisdiction.
It would be revealing to be able to compare a robopoll’s unadjusted and adjusted poll distributions in their pre-election preference measures. I suspect that if this information were available, it would reveal wide differences between the two estimates.
Because robopolls make subjective judgments when establishing their estimates of the composition of the likely electorate, this method can easily produce an entire array of different possible survey results. It is left to the robopollster to choose which result or political reality fits their own expectations at that moment in time. This is not only dangerous, it has the long-term effect of undermining public confidence in the objectivity of the entire public opinion polling process.
Concerns about the future of polling: As more pre-election polls employ the robopolling method, my fear is that they will crowd out the other higher quality polls that are being conducted, leaving the media and the public with a sometimes confusing batch of pre-election poll estimates to sort through.
This is not to say that better robopollsters are manipulating their poll results for their own ends. Some are trying to make up for the deficiencies in their initial survey samples. For example, at least one robopollster extended their data collection over a longer three-day period and experimented with the use of live interviewers to call separate samples of cell phone listings in an attempt to fill the gap of voters only reachable by cell phone.
Yet the other problems inherent in their survey approach remain. This is why I continue to view the results of most of them cautiously.
Polling on Prop. 19: One other controversy between robopolls and traditional telephone polls surfaced this year in California during the Prop. 19, marijuana legalization initiative campaign.
When polling this year on Prop. 19, the traditional telephone pollsters fielded a number of inquiries from reporters and others questioning the reliability of live interviewer telephone polls conducted on a controversial topic like marijuana. The theory they presented was that because robopolls avoid direct human interactions when conducting their polls, voters felt less constrained about admitting their true opinions on Prop. 19.
Most of the literature on interviewer effects on sensitive topics research like marijuana relate to people being asked about their own personal behaviors that might be embarrassing or socially undesirable. This, in my opinion, doesn’t apply when polling on a policy issue like Prop. 19, which simply asks voters their opinions about an initiative to legalize marijuana’s sale and use.
At the time, I challenged those questioning the accuracy of live interviewer polls on the topic to revisit the issue after the election. Well, the results are in and the live interviewer polls like Field, PPIC and the Times/USC polls were generally closer to the final vote on Prop. 19 than the robopolls.
In their final pre-election surveys the state’s three leading traditional telephone polls showed Prop. 19 trailing by an average of 8 percentage points. By contrast, the average of the two final pre-election robopolls conducted in California showed Prop. 19 trailing by just 4.5 percentage points. According to the California Secretary of State, with nearly nine million votes counted and more than one million votes yet to be counted, California voters were rejecting Prop. 19 by eight percentage points, 54% to 46%.
I hope this puts that theory to rest.
Mark DiCamillo is Senior Vice President, Field Research Corporation and Director of The Field Poll
