Lies, Damned Lies and Polls
The newest CBS News/NY Times Poll was released today. As with all polls, its results are reported by its sponsors as the definitive expression of the opinions of the American people as of this moment in time.
In the spirit of informed cynicism, let us go beyond the reassuring summaries provided by the Times and CBS and actually look at the pdf file of the results for all questions asked of the 1,052 people chosen, presumably at random, to represent the American people in toto.
All the way down on page 16 of the survey, at the top of the page, is a question of fact, not opinion, that can easily be tested for the veracity of the response obtained. It is:
Some people are registered to vote and others are not. Are you registered to vote in the precinct or election district where you now live, or aren't you?
Yes No
3/7-11/07 88 12
If 88 percent of these respondents assert that they are registered to vote, just how representative of the total population is this group? According to the Census Bureau, in the 2004 elections, the total number of persons eligible to vote was 215.7 million. Of those, only 142.1 million were registered to vote. That is 65.9%. Well short of the 88% who claim to be registered in the survey.
Are some of the respondents lying? Well, not to put too fine a point on it, yes. But it is a very human and understandable lie. A stranger calls you up and asks if you are, in essence, a good citizen as defined by the teachers and civic leaders of your childhood. The truth of the answer is unknowable to the questioner. Only a compulsively honest person would admit, or perhaps for some, gleefully acknowledge, their refusal to "go along" with the American civic expectation.
This kind of response bias is present in all random polling questions to one degree or another. It can be guaranteed to inflate the positive response to such a question. Here is another example from the current results:
68. How would you rate the financial situation in your household these days? Is it very good, fairly good, fairly bad or very bad?
Very good Fairly good Fairly bad Very bad DK/NA
11/10-16/88 12 69 13 5 2
10/8-10/90 14 70 11 4 1
7/21-25/06 12 64 17 5 1
4/20-24/06 (sic) 14 69 13 4 1
69. Think about your household income, would say that it is more than enough so that you can save and buy some extras, just enough to meet your bills and obligations, or it is not enough to meet your bills and obligations?
Can save and Just enough to pay Not enough to
buy extras bills and obligations pay bills etc. DK/NA
3/31-4/2/96 27 55 16 2
5/31-6/3/96 24 57 19 1
1/14-18/05 33 48 17 2
7/21-25/06 35 52 12 -
9/15-19/06 34 51 15 -
10/5-8/06 36 46 17 1
10/27-31/06 38 44 17 1
2/24-27/07 37 49 13 1
4/20-24/07 41 48 11 1
Although for April, 2007 in the first question a full 83% regard their financial situation as very good or fairly good, a large number in the second question, 48% claim to have just enough income to pay their bills, with no ability to purchase extras. I don't know about you, but for me, just meeting the cost of my bills does not define satisfaction. In such questions, the interpretation of the respondent is all important.
Here is a final example of the effect of the precise wording of any question:
56. In order to cut down on energy consumption and reduce global warming, would you favor or oppose an increased federal tax on gasoline?
Favor Oppose DK/NA
4/20-24/07 38 58 4
57. What if an increased tax on gasoline would reduce the United States' dependence on foreign oil, then would you favor or oppose an increased federal tax on gasoline?
ASKED OF HALF THE SAMPLE N=512
Favor Oppose DK/NA
2/22-26/06 55 37 8
4/20-24/07 64 30 6
In these two questions, asked one after the other, the introduction by the questioner of the notion that an increase in taxes on gasoline would reduce dependence on foreign oil makes the number in favor of such taxes jump from 38% to 64%. That is a 68% increase in the positive response.
My general point is the following: Oh, what a tangled web we weave when we seek to govern ourselves based on such a tenuous array of "facts". Pity (or perhaps condemn mercilessly) the poor politician who attempts to tailor his or her policy choices by such ephemeral data. The more I think about it, let's reserve any pity for the constituents of such fools.
In the spirit of informed cynicism, let us go beyond the reassuring summaries provided by the Times and CBS and actually look at the pdf file of the results for all questions asked of the 1,052 people chosen, presumably at random, to represent the American people in toto.
All the way down on page 16 of the survey, at the top of the page, is a question of fact, not opinion, that can easily be tested for the veracity of the response obtained. It is:
Some people are registered to vote and others are not. Are you registered to vote in the precinct or election district where you now live, or aren't you?
Yes No
3/7-11/07 88 12
If 88 percent of these respondents assert that they are registered to vote, just how representative of the total population is this group? According to the Census Bureau, in the 2004 elections, the total number of persons eligible to vote was 215.7 million. Of those, only 142.1 million were registered to vote. That is 65.9%. Well short of the 88% who claim to be registered in the survey.
Are some of the respondents lying? Well, not to put too fine a point on it, yes. But it is a very human and understandable lie. A stranger calls you up and asks if you are, in essence, a good citizen as defined by the teachers and civic leaders of your childhood. The truth of the answer is unknowable to the questioner. Only a compulsively honest person would admit, or perhaps for some, gleefully acknowledge, their refusal to "go along" with the American civic expectation.
This kind of response bias is present in all random polling questions to one degree or another. It can be guaranteed to inflate the positive response to such a question. Here is another example from the current results:
68. How would you rate the financial situation in your household these days? Is it very good, fairly good, fairly bad or very bad?
Very good Fairly good Fairly bad Very bad DK/NA
11/10-16/88 12 69 13 5 2
10/8-10/90 14 70 11 4 1
7/21-25/06 12 64 17 5 1
4/20-24/06 (sic) 14 69 13 4 1
69. Think about your household income, would say that it is more than enough so that you can save and buy some extras, just enough to meet your bills and obligations, or it is not enough to meet your bills and obligations?
Can save and Just enough to pay Not enough to
buy extras bills and obligations pay bills etc. DK/NA
3/31-4/2/96 27 55 16 2
5/31-6/3/96 24 57 19 1
1/14-18/05 33 48 17 2
7/21-25/06 35 52 12 -
9/15-19/06 34 51 15 -
10/5-8/06 36 46 17 1
10/27-31/06 38 44 17 1
2/24-27/07 37 49 13 1
4/20-24/07 41 48 11 1
Although for April, 2007 in the first question a full 83% regard their financial situation as very good or fairly good, a large number in the second question, 48% claim to have just enough income to pay their bills, with no ability to purchase extras. I don't know about you, but for me, just meeting the cost of my bills does not define satisfaction. In such questions, the interpretation of the respondent is all important.
Here is a final example of the effect of the precise wording of any question:
56. In order to cut down on energy consumption and reduce global warming, would you favor or oppose an increased federal tax on gasoline?
Favor Oppose DK/NA
4/20-24/07 38 58 4
57. What if an increased tax on gasoline would reduce the United States' dependence on foreign oil, then would you favor or oppose an increased federal tax on gasoline?
ASKED OF HALF THE SAMPLE N=512
Favor Oppose DK/NA
2/22-26/06 55 37 8
4/20-24/07 64 30 6
In these two questions, asked one after the other, the introduction by the questioner of the notion that an increase in taxes on gasoline would reduce dependence on foreign oil makes the number in favor of such taxes jump from 38% to 64%. That is a 68% increase in the positive response.
My general point is the following: Oh, what a tangled web we weave when we seek to govern ourselves based on such a tenuous array of "facts". Pity (or perhaps condemn mercilessly) the poor politician who attempts to tailor his or her policy choices by such ephemeral data. The more I think about it, let's reserve any pity for the constituents of such fools.
Labels: politicians, politics, polls
posted by Joe Hogan at
10:59 PM
