Friday, October 31, 2014


This is Joseph.

We’re all familiar with the critiques of standardized tests and other common measures used for high-stakes decisions.  Recently, somebody in my circle has started going on about measures of “grit” and their predictive power.  I am willing to believe that “grit” is an excellent predictor of all sorts of things.  But I wonder if much of the predictive power of “grit” comes from the fact that these measures are currently low-stakes, so people have few incentives to game them.
 I really think that this is the heart of the measurement problem.  Insofar as there is a way to do better on a test, in a way that is less work than just be really good at it, then it is probable that much of your signal will be gaming.  Studying the form of the question, for example, is likely to improve performance (by less confusion, if nothing else) but access to these approaches may vary by context.

Even worse, some of the test prep may have nothing to do with the underlying measure.  So the score starts to measure things like "willingness to sacrifice learning time for test prep time". 

This is a very good insight and likely to be eternally problematic in education. 

1 comment:

  1. The measurement problem that I pointed out there is only half the problem. If it were all of the problem I'd say we should just do away with standardized tests. But what are the other measures that one could use for admission to advanced educational programs?

    Grades have been critiqued from so many angles that one needn't dwell on the issues there, the limitations of "book smarts" just being one among many issues. (Though I will note that it is odd to hear educators dismiss success in school as a predictor of one's ability to succeed if admitted to an educational program.)

    There are lots of studies showing that people tend to write rec letters differently for men and women. Also, applicants from more advantaged backgrounds are more likely to have the sorts of experiences that enable them to get letters from more distinguished figures.

    Resumes can be padded, and even in their most honest form, the sorts of experiences that one has on the resume will be heavily influenced by one's level of advantage or disadvantage. Not to mention the cultural factors that play into what sorts of extracurricular activities one pursues.

    Essays can be improved with coaching, raising problems similar to standardized tests, students who use a dialect other than American Standard English at home will face certain disadvantages on essays, and how one expresses oneself in writing (and how an evaluator interprets written work) will be influenced by cultural background.

    I'm not a huge fan of standardized tests, but I've never heard a good answer for what the replacement should be if they are outright eliminated from all admissions decisions.