"Up until about a month ago, we all knew what merit pay meant."

Dana Goldstein has a good write-up on a recent education study:

In 10 cities, including Los Angeles, Miami, and Houston, researchers at Mathematica identified open positions in high-poverty schools with low test scores, where kids performed at just around the 30th percentile in both reading and math. To fill some of those positions, they selected from a special group of transfer teachers, all of whom had top 20 percent track records of improving student achievement at lower poverty schools within the districts, and had applied to earn $20,000 to switch jobs. The rest of the open positions were filled through the usual processes, in which principals select candidates from a regular applicant pool.

If a transfer teacher stayed in her new, tougher placement for two years, she’d earn the $20,000 in five installments, regardless of how well her new students performed. In public education, $20,000 is a whopping sum, far more generous than the typical merit pay bonus of a few hundred or a few thousand dollars.

In the process, a remarkable thing happened. The transfer teachers significantly outperformed control-group teachers in the elementary grades, raising student achievement by 4 to 10 percentile points—a big improvement in the world of education policy, where infinitesimal increases are often celebrated.

 but I have one big problem with the way she presents the findings.

That’s why the results of a new study, the Talent Transfer Initiative, financed by the federal government, are so important. Surprisingly, this experiment found merit pay can work.

Up until about a month ago, we all knew what merit pay meant. Under these systems of compensation, your pay would vary based on certain performance metrics from the previous period.

The initial rationale for applying this type of compensation to education was based on the assumption that teachers would work harder and do a better job if their performance (defined by their students' test scores) was tied to their pay . This played in very closely with the sub-narrative that problems in American education were largely driven by lazy, tenured teachers.

Unfortunately, when this idea was was tried in various pilot programs, it failed to show any substantial effect. It almost appeared as if teachers were already, for the most part, trying to do their best even when their pay didn't vary with performance.

This failure of concept led to a wave of revisionism from movement reformers such as Jonathan Chait. They quickly came up with a new claim. Under the revised history, rationale for merit pay had never been about incentives; push had always been about selection and retention of the best people.

There were always significant problems with this new rationale. Unless you were to assume that being a good teacher was strongly correlated with being bad at mathematics, the size of the bonuses would have to be very large to compensate for the deferment and variability of compensation. Even more troubling, the metrics proposed to evaluate teachers had been shown to be wildly unstable, thus teachers would have very little idea of what their annual take-home pay would be from year-to-year.

The flaws in that second rationale were so obvious, it almost suggested that the proponents were simply looking for a face-saving claim to make before backing away from the issue.

That face-saving quality is even more prominent in the coverage of the recent Chicago experiment in hiring bonuses for teachers. As generally presented by Goldstein and others, this seems like a partial win for both sides. Movement reformers can point to the findings and say that merit pay works while counter-reform advocates can point out that the teachers who got these great results were highly experienced and certified.

The problem with that story is that the first part of the claim is based on a complete 180-degree redefinition of merit pay. The bonus had nothing to do with metrics of success; instead it was solely contingent on those teachers serving out a specified term in the position. In other words, it used the exact same rationale that movement reformers have always objected to when applied to tenure and pay raises for seniority. (You can see Diane Ravitch making some of the same points here.)

I understand the benefits of keeping a debate civil. We should constantly make it clear that the vast majority of people on both sides of this debate share the common goal of improving education. We cannot, however, let the desire for civility become an excuse for dishonesty, particularly not when we are reporting on research that affects open policy questions.

This study suggest taking something like the Canadian approach to managing and compensating teachers. In other words, roughly the opposite of what those advocating merit pay have called for. You can question the quality of this study. You can call it impractical and question whether the results can be scaled up to a useful level. What you cannot do is claim that the data supports your position because you have changed your definitions mid-argument.