If you’re trying to distinguish between different students’ levels of understanding—particularly in situations where knowledge can possibly substitute for reasoning and comprehension—simply making questions harder will seldom help and will often do just the opposite. For example, if a Math Olympiad style test switched from geometry questions to trigonometry questions, the exam would mainly be good at identifying which students had taken pre-cal.
In these cases, a well-designed test will find a way of leveling the playing field so that additional information and training will not give one person an advantage over another. One of the best examples of this is the old SAT reasoning test, before David Coleman—The New York Times darling—“fixed” it.
An old English professor of mine (who, not entirely coincidentally, introduced me to Raymond Smullyan) accurately described it as the toughest ninth-grade math test you’ll ever take. In terms of knowledge, it didn’t require anything beyond Algebra I and a few really basic geometry concepts that were helpfully provided on the first page of the test. On top of that, forms of notation were invented so that the student who hadn’t taken a math course for a year or two was on a more or less equal playing field with the kid who was well into the first semester of calculus.
Back in 2014, we talked about how the SAT worked around the harder-problem fallacy (though not by that name) and about how the reporters covering the test (which was at the time out of fashion with the NYT et al. before shifting again) kept missing the point.
As you have guessed, we'll be connecting this to our AI thread in a few days.
Perhaps we should add "opaque" to the list of journalists' vocabulary questions
Last week, Andrew Gelman criticized Todd Balf for picking words and phrases for their emotional connotation rather than for their actual meaning in his New York Times Magazine article
on the changes in the SAT. 'Jeffersonian' was the specific term that
Gelman choked on. I'd add 'opaque' to the list though the blame here
mainly goes to David Coleman, president of the College Board and quite
possibly the most powerful figure in the education reform movement:
For the College Board to be a great institution, [Coleman] thought at the time, it had to own up to its vulnerabilities. ... “It is a problem that it’s opaque to students what’s on the exam."There's a double irony here. First because Coleman has been a long-standing champion of some very opaque processes, notably including those involving standardized tests, and second because test makers who routinely publish their old tests and who try to keep those tests as consistent as possible from year to year are, by definition, being transparent.
This leads to yet another irony: though the contents of the tests are readily available, almost none of the countless articles on the SAT specifically mention anything on the test. The one exception I can think of is the recent piece by Jennifer Finney Boylan, and it's worth noting that the specific topic she mentioned isn't actually on the test.
Being just a lowly blogger, I am allowed a little leeway with journalistic standards, so I'm going to break with tradition and talk about what's actually on the math section of the SAT.
Before we get to the questions, I want to make a quick point about geometry on the SAT. I've heard people argue that high school geometry is a prerequisite for the SAT. I don't buy that. Taking the course certainly doesn't hurt, but the kind of questions you'll see on the exam are based on very basic geometry concepts which students should have encountered before they got to high school. With one or two extremely intuitive exceptions, all the formulas you need for the test are given in a small box at the top of the first page.
As you are going through these questions, keep in mind that you don't have to score all that high. 75% is a good score. 90% is a great one.
You'll hear a lot about trick questions on the SAT. Most of this comes from the test's deliberate avoidance of straightforward algorithm questions. Algorithm mastery is always merely an intermediary step -- we care about it only because it's often a necessary step in problem solving (and as George Pólya observed, if you understand the problem you can always find someone to do the math) -- but when students are used to being told to factor this and simplify that, being instead asked to solve a problem, even when the algorithms involved are very simple, can seem tricky and even unfair.
There are some other aspects of the test that contribute to the reputation for trickiness:
Questions are written to be read in their entirety. One common form breaks the question into two parts where the first part uses a variable in an equation and the second asks the value of a term based on that variable. It's a simple change but it does a good job distinguishing those who understand the problem from those who are merely doing Pavlovian mathematics where the stimulus is a word or symbol and the response is the corresponding algorithm;
Word problems are also extensively used. Sometimes the two-part form mentioned above is stated as a word problem;
One technique that very probably would strike most people as 'tricky' actually serves to increase the fairness of the test, the use of newly-minted notation. In the example below, use of standard function notation would give an unfair advantage to students who had taken more advanced math courses.
One thing that jumps out when us math types is how simple the algebraic concepts used are. The only polynomial factoring you are ever likely to see on the SAT is the difference between two squares.
A good grasp of exponents will also be required for a perfect score.
I've thrown in a few more to make it a more representative sample.
We can and should have lots of discussions about the particulars here -- I'm definitely planning a post on Pavlovian mathematics (simple stimulus/algorithmic response) -- but for now I just want to squeeze in one quick point:
Whatever the SAT's faults may be, opaqueness is not among them. Unlike most of the instruments used in our metric-crazed education system, both this test and the process that generates it are highly transparent. That's a standard that we ought to start extending to other tests as well.















