After disruption, though, there comes at least some version of stage three: overshoot. The most common problem is that all these new systems—metrics, algorithms, automated decision making processes—result in humans gaming the system in rational but often unpredictable ways. Sociologist Donald T. Campbell noted this dynamic back in the ’70s, when he articulated what’s come to be known as Campbell’s law: “The more any quantitative social indicator is used for social decision-making,” he wrote, “the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
On a managerial level, once the quants come into an industry and disrupt it, they often don’t know when to stop. They tend not to have decades of institutional knowledge about the field in which they have found themselves. And once they’re empowered, quants tend to create systems that favor something pretty close to cheating. As soon as managers pick a numerical metric as a way to measure whether they’re achieving their desired outcome, everybody starts maximizing that metric rather than doing the rest of their job—just as Campbell’s law predicts.This really explains two phenomenon that are tightly linked. One, how quants can initially get really great results. But, over time, when people understand how the derived numbers come from we get the second part: these system seem to crash rather badly. After all, initially, a metric like number of arrests for police officers might be a proxy for diligence. But, if it is all that one uses, it may end up being the reverse of what is intended (a shirker could focus on that metric, figuring if they are rated as a top performer then other performance issues won't matter).
The link between this phenomenon and high stakes teacher testing should be rather obvious.
As an aside, I think the blogger team here continues to disagree with Felix on media issues. But sometimes it is worth remembering that he is worth engaging because the median post is so good that the others stand out. The price of good work?