A subtle issue with standardized tests

This is Joseph.

Dean Dad has a nice piece on assessment.  A part of it that jumped out was:

Johnson’s argument is subtle enough that most commenters seemed to miss it.  In a nutshell, he argues that subjecting existing instruction to the assessment cycle will, by design, change the instruction itself.  Much of the faculty resistance to assessment comes from a sense of threatened autonomy.  Johnson addresses political science specifically, noting that it’s particularly difficult to come up with content-neutral measures in a field without much internal consensus, and with factions that barely speak to each other. 

He’s right, though it may be easier to grasp the point when applied to, say, history.  There’s no single “Intro to History” that most would agree on; each class is the history of something.  The ‘something’ could be a country, a region, a technology, an idea, an art form, or any number of other things, but it has to be something specific.  Judging a historian of China on her knowledge of colonial America would be easy enough, but wouldn’t tell you much of value.  If a history department finds itself judged on “scores” based on a test of the history of colonial America, then it can either resign itself to lousy scores or teach to the test.

This means that the design of standardized test is crucially important if students and/or teachers are going to be evaluated on them.  For some subjects, e.g. basic math, this may be less controversial but it still involves making choices about what the emphasis will be.  A perfect test is like a perfect teachers -- neither beast really exists in nature. 

But this is critically important for high stakes tests, because what is taught cannot help but be influenced by the test.  If history questions on the high stakes tests are all focused on colonial America, guess what the history section of classes will look like.  In some sense that is okay, insofar as we have a broad consensus as to what should be taught.  But it does make the content of the tests a matter of public policy and concern as much as any other aspect of school instruction.