Testing teachers

It has been a while since we talked about education reform but Arnold Kling has a great perspective:

Simple formulas can be “gamed.” That is, employees learn to achieve the objectives in the formula while failing to work toward the longer-term goals of the firm. On Wall Street, we have seen how bonus formulas proved dysfunctional. The older partnership form of organization appears to have provided better incentives.

A government-run system of teacher compensation, based on test scores, would in some ways be the worst of all worlds. It would create incentives for teachers to “game” the system. It would give too much weight to a noisy indicator of performance. As a result, it would do little or nothing to improve accountability or to reward better teachers.

This classic insight is shown in this dilbert comic.

Are we sure that test score based measures are the way to go? Most information technology jobs have the same sort of issues, often solved by comparative rankings and broad evaluations. Even worse, a bad metric drives out the good (meaning it could actually be counter-productive).

The whole post is worth reading.

Performance based teaching

Via Megan McArdle, comes this gem:

It would be naïve to assume that the persons subjected to variable pay-for-performance would accept the respective criteria in a passive way and fulfil their work accordingly. Rather, they spend much energy and time trying to manipulate these criteria in their favour. This is facilitated by the fact that employees often know the specific features of their work better than their superiors. The wage explosions observable in many sectors of the economy can at least partly be attributed to such manipulations, eg when managers are able to contract easily achievable performance goals.

Arnold Kling goes on to link this with education reform:

When a remote authority sets incentives, people respond by manipulating the system. This fact is poorly understood by education reformers who are fond of pay-for-performance and national standards, by health care reformers who are fond of paying for quality, and by financial regulators. In fact, the quoted paragraph provides an excellent description of the financial regulatory process under risk-based capital. The banks spent much energy and time trying to manipulate the risk-based capital regulations in their favor. They got what they wanted, in terms of risky portfolios backed by little capital.

The Hayekian story here is that effective compensation practices require local knowledge and tacit knowledge. In a large company, you give a middle manager a fair amount of discretion in compensating his or her staff. If instead you try to implement an automated bonus system, you will get gamed.

I think that this hits at the heart of the concerns Mark has been expressing for a while. Test-based systems that are implemented at a very high level encourage all sorts of behavior, and it is quite possible that manipulating the system will be easier than actually improving performance. Even worse, bad performers (gaming the system) have an advantage over good performers as they can get top scores for less total effort.

Bed performance drives out good performance and things get worse. I think Megan is very right to be skeptical about how easy it is to reform systems once they adopt this management style as the new backbone of the labor force are the people who thrive in gaming the system and they will resist change back to older approaches.

I wish we'd look more at the history of countries like Russia for how difficult it is to make top-down reform and economic control work at the national level.

Arnold Kling

Arnold Kling is a skeptic about the importance of peer effects:

Back when the SAT was just math and verbal, I described us as living on the other side of a 250-point SAT gradient--in the better high schools, the SAT's were 1200, and in our high school they were 950. (I took the view that my own kids did not need high-achieving peers in order to do well on their own.)

I am not sure whether this is persuasive (although it does match my personal experience). But it is true that this is the sort of behavior that I would expect from peer-effects skeptics (and if we saw more of it there would be a much smaller premium on good schools for housing costs.