Discrimination in Standardized Tests

One complaint you often hear about standardized tests is that they’re racist or sexist. The complaint is usually framed this way:

The average score of group X is higher than the average score of group Y, so the test must be biased against group Y in favor of group X.

This argument is used to show that standardized tests are skewed in favor of men over women, or in favor of whites over blacks and latinos, or in favor of asians over whites, or in favor of northerners over southerners, or who knows what. (Here’s an example of the argument being applied to the “gender gap” on the SAT, from FairTest.org: http://www.fairtest.org/facts/genderbias.htm)

It’s certainly true that a standardized test can be racist or sexist by design. It’s also true that previous versions of the SAT, in particular, included racist and sexist test items.

But the major standardized tests have been changed many times in the past few decades, and no credible argument can be made that the tests are still racist or sexist in themselves.

First, let’s take a look at the argument I described above—the idea that the over- or under-representation of a certain group means that the selection process is biased. This idea crumbles pretty quickly if we look at it closely.

FairTest says the SAT is biased in favor of men because men have a higher average score on the test, even though women have higher average GPAs. But why not look at the same data from another direction and conclude that high school is simply biased in favor of women? It makes just as much sense to draw this conclusion from a comparison of SAT and GPA data, but anybody who’s ever been to high school knows it would be ridiculous to claim that high school is biased against men. Similarly, anyone who actually looks at the SAT would have to conclude that it demonstrates no bias.

Let’s look at the same ridiculous argument in another context. Roughly 40% of the welfare recipients in America are black, even though black people only make up about 10% of the national population. Should we conclude from this over-representation that the welfare system is biased in favor of blacks? Of course not—but that’s exactly the conclusion that Fairtest’s “logic” would require.

Don’t get me wrong—it’s very possible to create a standardized test that discriminates against a particular group. All you’d have to do is include questions that assume a kind of knowledge that members of the group might not have. For example, imagine a test question about all the possible permutations of the digits in a phone number. In order to answer the question, you’d need to know that American phone numbers have 10 digits. Such a question would be biased against people who aren’t familiar with the American telephone system. An analogy like “ANALECTS : CONFUCIUS :: dialogues : Plato” would be biased in favor of people familiar with Chinese and Greek history. And so on—you get the idea.

(For what it’s worth, neither of the questions I just described could ever appear on the SAT, PSAT, GRE, LSAT, or GMAT, because they violate the rules and patterns of these tests—which is exactly the point I’m trying to make.)

So it would definitely be possible to create a biased standardized test. In fact, there’s good evidence to suggest that earlier versions of today’s standardized tests suffered from such biases. But the important question for us, and for groups like FairTest, is whether such biases still exist today. An impartial review of test items from the major standardized tests can only conclude that there is no longer any race or gender bias present in them.

In my career as a test-prep guy, I’ve looked at more test items than I could ever hope to count. Here’s the most biased SAT question I’ve ever run across:

The incomplete table above categorizes the members of a chess club according to their age and status. During a tournament, each member of the club plays exactly one game with each of the other members. How many games of chess are played between amateurs 20 years or older and professionals under 20 years old in the tournament?

This question is number 21 on page 327 of the out-of-print College Board book 10 Real SATs. It’s biased because it assumes that the reader knows a chess game involves two players. (I should also add that the answer choices all assume only two players per game, so you would still be able to work out the solution even if you don’t know how chess is played.) I don’t know who this bias would work against, from a racial/gender standpoint, because I don’t know which races and genders play the most high school chess, but there you have it.

But all of this begs a question: If the major standardized tests aren’t inherently biased, then what accounts for the reported disparity in the average scores of different groups?

Unfortunately, we don’t have the data necessary for a good answer. I have my theories, of course. I think the major factor contributing to the average score differences among various racial groups has to do with the quality of public education. The racial groups with lower scores on standardized tests generally get the worst the public school system has to offer in the formative K-12 years. And how do we account for the difference between the genders? That’s harder to answer, but not impossible. First of all, more female test-takers than male test-takers come from lower-scoring minorities; this means that the racial disparities (which, again, are probably the result of unequal access to public education, not inherent testing biases) are skewing the gender-based data. It should also be noted that the difference between male and female test-takers is within the SAT’s admitted 50-point margin of error, while the difference between white and black test-takers was 186 points in 1989 (source: William Bowen and Derek Curtis. The Shape of the River: Long-Term Consequences of Considering Race in College and University Admissions. 1998).

Don’t take my word for it, though—get out some copies of real test questions and see for yourself. If you find biased test items, I hope you’ll post page references in the comments, with an explanation of the bias you identify in the question.