If you’re trying to distinguish between different students’ levels of understanding—particularly in situations where knowledge can possibly substitute for reasoning and comprehension—simply making questions harder will seldom help and will often do just the opposite. For example, if a Math Olympiad style test switched from geometry questions to trigonometry questions, the exam would mainly be good at identifying which students had taken pre-cal.
In these cases, a well-designed test will find a way of leveling the playing field so that additional information and training will not give one person an advantage over another. One of the best examples of this is the old SAT reasoning test, before David Coleman—The New York Times darling—“fixed” it.
An old English professor of mine (who, not entirely coincidentally, introduced me to Raymond Smullyan) accurately described it as the toughest ninth-grade math test you’ll ever take. In terms of knowledge, it didn’t require anything beyond Algebra I and a few really basic geometry concepts that were helpfully provided on the first page of the test. On top of that, forms of notation were invented so that the student who hadn’t taken a math course for a year or two was on a more or less equal playing field with the kid who was well into the first semester of calculus.
Back in 2014, we talked about how the SAT worked around the harder-problem fallacy (though not by that name) and about how the reporters covering the test (which was at the time out of fashion with the NYT et al. before shifting again) kept missing the point.
As you have guessed, we'll be connecting this to our AI thread in a few days.
Perhaps we should add "opaque" to the list of journalists' vocabulary questions
Last week, Andrew Gelman criticized Todd Balf for picking words and phrases for their emotional connotation rather than for their actual meaning in his New York Times Magazine article
on the changes in the SAT. 'Jeffersonian' was the specific term that
Gelman choked on. I'd add 'opaque' to the list though the blame here
mainly goes to David Coleman, president of the College Board and quite
possibly the most powerful figure in the education reform movement:
For the College Board to be a great institution, [Coleman] thought at
the time, it had to own up to its vulnerabilities. ... “It is a problem
that it’s opaque to students what’s on the exam."
There's a double irony here. First because Coleman has been a
long-standing champion of some very opaque processes, notably including
those involving
standardized tests,
and second because test makers who routinely publish their old tests
and who try to keep those tests as consistent as possible from year to
year are, by definition, being transparent.
This leads to yet another irony: though the contents of the tests are
readily available, almost none of the countless articles on the SAT
specifically mention anything on the test. The one exception I can think
of is the recent
piece by Jennifer Finney Boylan, and it's worth noting that the specific topic she mentioned
isn't actually on the test.
Being just a lowly blogger, I am allowed a little leeway with
journalistic standards, so I'm going to break with tradition and talk
about what's actually on the math section of the SAT.
Before we get to the questions, I want to make a quick point about
geometry on the SAT. I've heard people argue that high school geometry
is a prerequisite for the SAT. I don't buy that. Taking the course
certainly doesn't hurt, but the kind of questions you'll see on the exam
are based on very basic geometry concepts which students should have
encountered before they got to high school. With one or two extremely
intuitive exceptions, all the formulas you need for the test are given
in a small box at the top of the first page.
As you are going through these questions, keep in mind that you don't
have to score all that high. 75% is a good score. 90% is a great one.
You'll hear a lot about trick questions on the SAT. Most of this comes
from the test's deliberate avoidance of straightforward algorithm
questions. Algorithm mastery is always merely an intermediary step -- we
care about it only because it's often a necessary step in problem
solving (and as
George Pólya observed,
if you understand the problem you can always find someone to do the
math) -- but when students are used to being told to factor this and
simplify that, being instead asked to solve a problem, even when the
algorithms involved are very simple, can seem tricky and even unfair.
There are some other aspects of the test that contribute to the reputation for trickiness:
Questions are written to be read in their entirety. One common form
breaks the question into two parts where the first part uses a variable
in an equation and the second asks the value of a term based on that
variable. It's a simple change but it does a good job distinguishing
those who understand the problem from those who are merely doing
Pavlovian mathematics where the stimulus is a word or symbol and the
response is the corresponding algorithm;
Word problems are also extensively used. Sometimes the two-part form mentioned above is stated as a word problem;
One technique that very probably would strike most people as 'tricky'
actually serves to increase the fairness of the test, the use of
newly-minted notation. In the example below, use of standard function
notation would give an unfair advantage to students who had taken more
advanced math courses.
One thing that jumps out when us math types is how simple the algebraic
concepts used are. The only polynomial factoring you are ever likely to
see on the SAT is the difference between two squares.
A basic understanding of the properties of real numbers is required to answer many of the problems.
A good grasp of exponents will also be required for a perfect score.
There will be a few problems in basic statistics and probability:
I've thrown in a few more to make it a more representative sample.
We can and should have lots of discussions about the particulars here --
I'm definitely planning a post on Pavlovian mathematics (simple
stimulus/algorithmic response) -- but for now I just want to squeeze in
one quick point:
Whatever the SAT's faults may be, opaqueness is not among them. Unlike
most of the instruments used in our metric-crazed education system, both
this test and the process that generates it are highly transparent.
That's a standard that we ought to start extending to other tests as
well.