Keep The SAT

Today I had originally planned to do a post on guessing strategies for the LSAT to complement my earlier article on guessing strategies for the SAT. But an article that appeared on American.com on Friday has forced me to pre-empt that post.

Charles Murray’s article is titled Abolish The SAT, and that pretty much says it all. You may remember Murray’s name from The Bell Curve, the 1994 book that he co-authored with Richard Herrnstein in which he incorrectly concluded, among other things, that blacks tend to be intellectually inferior to whites for reasons of genetics.

Murray’s argument for an end to the SAT is just as flawed as his arguments on race and intelligence. He relies on his and others’ statistical analyses of data on thousands of students’ performances. This will undoubtedly impress the type of reader who is easily persuaded by the number of citations in an article. But citations alone don’t make you right.

For the sake of clarity, I’ll address Murray’s arguments in increasing order of importance, as opposed to addressing them in the order he made them. Among other things, that lets me get Murray’s accusations against the test-prep industry out of the way early, so we can focus on more important issues. I’ll also assume that all of the statistical evidence Murray cites is valid and current, as opposed to analyzing the methodologies used to gather that evidence or pointing out that some of it is over 10 years old; even if we assume that Murray’s information is accurate, his conclusions are indefensible.

Let’s get started . . .

“A few dozen points:” What’s test prep good for, anyhow?

Murray mentions several studies that found that, “on average, coaching raises scores by no more than a few dozen points, enough to sway college admissions in exceedingly few cases.” Further, he says that Kaplan and Princeton Review, the two major SAT-prep companies, can provide no evidence of widespread success achieving 100- and 200-point increases. From this, Murray concludes that coaching has no real effect on SAT scores.

Murray doesn’t seem to realize that talking about “average” gains on the SAT is meaningless, because college applications aren’t reviewed in the aggregate. What matters is whether individual students experience significant score gains as a result of coaching, and Murray’s statistics don’t tell us anything about whether this is happening. Imagine that half of the people who receive SAT coaching see score increases of 1000 points, while the other half see decreases of 980 points. The resulting average score change would be 10 points, which sounds inconsequential, but the individual score changes for any single test-taker would be huge.

That hypothetical situation is a little extreme, of course, but the point here is that Murray’s “average coaching increase” is an irrelevant number. A significant portion of the students who get coaching might still be seeing significant increases.

Or, to use an apt simile, saying that you shouldn’t use a coach for the SAT because the average score increase is insignificant is like saying that your basketball team shouldn’t practice because the average team wins exactly half of its games: Average results for a group are meaningless in situations where individual experiences are what matter.

Murray also makes the mistake of believing, as many consumers do, that the best test preparation must come from the largest test-prep companies. Even if it turns out that Kaplan and Princeton Review don’t usually get their students large score increases, that wouldn’t prove that test prep itself is doomed to fail. My own experiences with satisfied customers, their unsolicited happy emails, and word-of-mouth advertising proves that effective test prep does exist, and that some test-takers do avail themselves of it.

So here’s another simile: Saying that test prep must have no effect because Kaplan and Princeton Review are bad at it is like saying that exercise must have no effect because the Ab Roller is a bad product.

(If you’d like to see an example of what good, effective test prep involves, please download my test prep White Paper.)

“Predictive validity:” So what?

Murray says that the SAT has no bias against any ethnic, social, or gender group because of a statistical test of its predictive validity. Basically, Murray says that if the SAT is unbiased, then it should predict everyone’s performance in college with equal precision regardless of race, gender, or socioeconomic status.

Murray reports that the current SAT satisfies this requirement, and concludes that the test must not be biased. (Actually, Murray finds the test to be slightly biased in favor of black students, because he says it predicts that they’ll do a little bit better in college than they actually do. At any rate, he doesn’t find the SAT to be biased against anyone.)

While I agree that the SAT is unbiased, the notion of predictive validity as a necessary indicator of bias is absurd because, among other things, it assumes that the college environment is also unbiased, which may or may not be the case. If the college environment is biased, then a test with perfect predictive validity for that biased environment could also be biased.

We can only tell if the SAT is biased by looking directly at the questions on it. If they don’t require any outside knowledge specific to the experience of any particular group, then the questions can’t be biased. If they do require knowledge unique to a specific group, then the questions are biased. The average performances of members of different groups are irrelevant to this analysis.

“One-tenth of a percentage point:” Keeping it real

Murray’s major complaint about the SAT is that it adds “just one-tenth of a percentage point to the percentage of variance in freshman grades explained by high school grade point and the achievement tests.” In other words, it doesn’t add anything to the average predictive value of a combination of GPA and achievement test scores.

Let’s assume, for Murray’s sake, that the predictive value of the SAT is currently exactly the same as a combination of GPA and other test scores for every single student. In other words, let’s waive the objection that, in many cases, individual students do significantly better or worse on one of those three indicators than they do on the other two, and let’s pretend that such disparities can never be telling for admissions committees.

Even if the SAT doesn’t add anything to the overall picture of an admissions candidate, it still serves the purpose of calibrating the other measures against a standard. The SAT is the objective yardstick that assures us that GPA and achievement test indicators, which are based on different standards for different students, remain meaningful.

Or, to use yet another simile, saying that the SAT is unnecessary because it generally re-confirms other indicators is like saying that the Supreme Court is unnecessary because the other two branches of government generally follow the law: Without an external restraint, there would be no way to know for certain that people were still playing by the rules. We’ll get into more of that below.

“Cram courses for US history:” What does test-prep actually entail?

Murray says that standardized achievement tests aren’t coachable because they stress knowledge of subject matter over abstract cognitive ability. He sees this as a good thing, because he thinks it would force the test-prep industry to focus on subject areas like science and history. He also says that students wouldn’t need test-prep companies anymore, since they can learn history, chemistry, and other subjects on their own.

One of the most obvious ways to refute this idea is to look at the existing standardized achievement tests. There are already various SAT IIs, AP exams, and IB tests covering every subject from Chinese to Psychology. Every single one of these tests can be prepped for in a non-obvious way that will raise scores significantly in a short time. Although Murray may not realize it, it’s impossible to write a standardized test that a good test-prep coach can’t crack. Standardized tests have to follow a set of rules by definition, and it will always be possible to infer what those rules are by analyzing past test items.

As an example, I was able to score a 3 on my AP Chemistry exam by learning only two formulas, even though I didn’t really know how to apply them properly. I just put the numbers from every single question into those formulas and looked to see if the result was one of the answer choices. If it was, I marked it. My score wasn’t great, but it got me out of taking chemistry in college. I achieved the same result as many, many students who actually tried to learn chemistry in order to get ready for the AP Chemistry test. With a more formalized and refined approach, I could have done even better (at the time I knew very little about standardized tests).

You might point out that most people don’t try to game subject-matter tests like they game the SAT, but this is only because subject-matter tests aren’t thought to be used in the admissions process the way Murray says they should be. If subject-matter tests became as important as the SAT, test-prep guys like me would be spending much more time looking for ways to beat them.

You can see an example of what might happen to those tests when you look at the multiple choice section on the SAT Writing exam. Those multiple-choice questions are supposed to test a student’s knowledge of grammar, and it’s true that most people who prep for them try to learn grammar out of a book. But the most effective way to prepare for those questions involves honing skills like measuring the length of the answer choices and learning to look for words that end in –ed or –ing, none of which has anything to do with actually learning grammar. We know this because we’ve looked at huge numbers of real SAT questions and figured out what appears on them all the time. We’ll be able to do the same thing with any standardized test (again, check out the white paper for examples of fast, non-obvious solutions to real questions from standardized tests).

What would happen if we followed Murray’s recommendations?

Murray says we should get rid of the SAT and switch to an admissions system in which students are judged only on their grades and their scores on standardized achievement tests. Unfortunately, he fails to anticipate the ramifications of that change.

As I mentioned above, Murray finds that the SAT adds practically nothing to the predictive value of a combination of GPA and achievement test scores. He says nothing at all, though, about what will happen to grades when they become a significant factor in the admissions game without the SAT to temper them.

It isn’t hard to imagine that the more responsive high schools (that is, private schools and high-end public schools with active parent communities) will inflate their grades even more than they already do once the SAT isn’t around anymore. This will give their students higher GPAs and make them even more competitive. Meanwhile, students who attend less responsive high schools (that is, schools with less competent faculties and virtually no parental involvement) won’t benefit as much from grade inflation. The result will be a widening gap between the GPAs at wealthier schools and the GPAs at poorer schools—the same sort of problem that Murray says the SAT was originally designed to combat.

How do we know this will happen? All we have to do is look at law schools and business schools. Because suitability for internships and jobs in those fields is largely determined by student GPAs, and because applicants want the best possible chances of getting into the highest-quality positions, many competitive professional schools inflate grades openly by setting and maintaining strict curves. The same kinds of things will happen with high schools once GPA becomes more important in the undergraduate admissions process.

This is one of the most frustrating things about Murray’s flawed analysis: He realizes that the SAT (or ACT) is the only feature common to every applicant, and he acknowledges that the major problem with the SAT is a public perception that higher SAT scores can be bought. Fine. Inexplicably, his solution to this problem is to abolish the commonality of the SAT and place more emphasis on GPA, the single measurement that can be purchased outright with a private school tuition or negotiated upwards by involved parents.

At any rate, those are my thoughts—what are yours? Leave a comment and let me know.