A Case Study in How NOT to Conduct Informative Surveys

I'm taking a break from philosophy of religion issues to briefly comment on a different issue that recently caught my attention. Next week, when I have more time, I'll put up the promised post on philosophy of mind issues.

Recently, a bit of a splash was made by the reported results of a survey conducted by economists Zeljka Buturovic and Daniel Klein. The survey asked several thousand non-economists to indicate whether they agreed or disagreed with eight economic statements. They were also asked to identify their political leanings. The purported aim of the survey was to correlate lay economic knowledge (that is, knowledge among non-economists) with political leanings—and the reported result was that liberals are in general far less knowledgeable than conservatives on these matters.


The survey measured, not the frequency of correct responses, but the frequency of incorrect ones (in other words, “Not sure” results were treated the same as correct responses). A response was judged incorrect if the respondent “somewhat” or “strongly” disagreed with a statement regarded as true by economists, or if they “somewhat” or “strongly” agreed with statements regarded as false (only statements enjoying a broad consensus among professional economists were used). The result? The more conservative you identified yourself as being, the fewer you got wrong. The more liberal you identified yourself as being, the more you got wrong.

Well, this result immediately made me suspicious—as it did to ianstoner, a contributor to The Philosopher’s Eye, who pointed out in a short blog post the substantial gap between the narrow technical meanings that economists attach to many of the words used in the survey and the more normative meanings that lay people attach. He paid particular attention to one of the statements in the survey, which reads as follows:

“Third World workers working for American companies overseas are being exploited.”

According to economists, this statement is false. Why? Because Third World workers employed by American companies overseas are economically better off on average than their peers. When economists say there isn’t exploitation taking place, that is all they mean. But ianstoner notes that this decidedly isn’t all that most of us mean when we use a term like “exploited,” since in ordinary usage “you can’t weigh in on this proposition without taking a stand on the standards of living that people in developing countries deserve when they’re working for American corporations.” In other words, if you think these workers are getting less than they deserve, you might be inclined to say they are being exploited even if you know full well that they are better off than the impoverished masses who are digging around in garbage heaps for some useful scrap they can barter for food.

In fact, however, “exploitation” has even more meanings than this. While some would define exploitation in the manner ianstoner favors, I tend to construe exploitation as taking advantage of someone’s desperate circumstances. Imagine someone who goes into a country afflicted with desperate poverty and, seeing an opportunity for personal gratification, offers money and food to a desperate young mother without the resources to feed her infant—if she agrees to have sex with him. Has he exploited her? If, by “exploitation,” you mean “taking advantage of someone’s desperation to get them to serve your desires,” then he has absolutely exploited her. And this remains true even if she ends up better off economically than those starving mothers who didn’t have the “good luck” to come across lascivious foreigners eager to make use of them.

In any event, this sample survey question got me interested in knowing what else was included in the survey. So here are the eight statements respondents were asked to assess:

1) Mandatory licensing of professional services increases the prices of those services (true according to economists). 2) Overall, the standard of living is higher today than it was 30 years ago (false). 3) Rent control leads to housing shortages (true). 4) A company with the largest market share is a monopoly (false). 5) Third World workers working for American companies overseas are being exploited (false). 6) Free trade leads to unemployment (false). 7) Minimum wage laws raise unemployment (true). 8) Restrictions on housing development make housing less affordable (true).

In looking at these statements, I see an overall trend. In general, those who self-identify as liberal tend to be more suspicious of unregulated markets. They worry about the ways in which businesses might take advantage of ordinary people who need the jobs, goods, and services that businesses provide, as well as about the effects of unregulated business practices on the environment. For these reasons they tend to favor greater government oversight and regulation of business practices.

Conservatives, by contrast, tend to trust the free market more and the government less, and so are presumptively suspicious of government regulation.

But here’s the problem. If you look at the eight statements above, you find that when some kind of regulation of business is featured in a “true” statement, it is correlated with a negative economic fact. When it is featured in a “false” statement, it is correlated with a positive economic fact. In other words, the statements are structured so as to appeal to the prejudices of conservatives while grating against the prejudices of liberals. Perhaps one reason why conservatives scored higher than liberals is because a majority of the questions were geared to appeal to conservative biases. It wasn’t that the conservatives “knew” some economic fact that the liberals didn’t know. Rather, it was that the conservatives liked the true statements and disliked the false one, while it was the reverse for the liberals.

Of course, as one of the survey’s creators, Daniel Klein, has noted, not all the survey questions fit this account. Take, for example, number 2: “Overall, the standard of living is higher today than it was 30 years ago” (a purportedly true statement). Here, it may be worth knowing when the poll was administered: December 2008. George Bush was still in office and the economy was in crisis. Liberals were inclined to say, “See what the Republican policies of deregulation have done? Our country has been driven to the brink of ruin!” Republicans were inclined to be more defensive of the current state of affairs. Again, pre-existing biases might lead liberals and conservatives to make different assessments of this statement regardless of how much or little they “know” about the economy.

Of course, there’s also the question of what is meant by “standard of living.” Economists will give this term a strictly monetary meaning, but the general population will have a vaguer understanding in terms of broad life values. If liberals and conservatives have different values, they’ll likely reach different answers even if they agree on the facts.

As to question #4, this is about the technical meaning of “monopoly.” Klein noted that liberals were more likely to get this wrong than conservatives. But what does that show? That conservatives have broader economic knowledge? Or that conservatives were more likely to enjoy playing Monopoly as kids, while their more liberal friends were playing Scrabble?

In any event, it is clear that this is a pretty lousy study, and that nobody should, on its basis, conclude that conservatives know more about economics than do liberals.