After disruption, though, there comes at least some version of stage three: overshoot. The most common problem is that all these new systems—metrics, algorithms, automated decision making processes—result in humans gaming the system in rational but often unpredictable ways. Sociologist Donald T. Campbell noted this dynamic back in the ’70s, when he articulated what’s come to be known as Campbell’s law: “The more any quantitative social indicator is used for social decision-making,” he wrote, “the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
On a managerial level, once the quants come into an industry and disrupt it, they often don’t know when to stop. They tend not to have decades of institutional knowledge about the field in which they have found themselves. And once they’re empowered, quants tend to create systems that favor something pretty close to cheating. As soon as managers pick a numerical metric as a way to measure whether they’re achieving their desired outcome, everybody starts maximizing that metric rather than doing the rest of their job—just as Campbell’s law predicts.This really explains two phenomenon that are tightly linked. One, how quants can initially get really great results. But, over time, when people understand how the derived numbers come from we get the second part: these system seem to crash rather badly. After all, initially, a metric like number of arrests for police officers might be a proxy for diligence. But, if it is all that one uses, it may end up being the reverse of what is intended (a shirker could focus on that metric, figuring if they are rated as a top performer then other performance issues won't matter).
The link between this phenomenon and high stakes teacher testing should be rather obvious.
As an aside, I think the blogger team here continues to disagree with Felix on media issues. But sometimes it is worth remembering that he is worth engaging because the median post is so good that the others stand out. The price of good work?
Joseph:
ReplyDeleteRegarding your disagreements with Felix on media issues (you're talking about rabbit ears and Netflix here?): Of course one possibility is that he's right and you're wrong. But, assuming that you're right and he's wrong, one explanation is that, because he's a big name and you are obscure, it's easy for him to dismiss your remarks. It can be difficult to have a good exchange of views when there is a power differential, even when both parties are generally well-intentioned and open-minded.
Hi Andrew,
ReplyDeleteI might have mis-stated my views a bit above. I was trying to pay Felix a very high compliment (his work is so tight that when he does a post I disagree with it jumps out). Of course, it's totally possible that he is more right than me -- I am happy to have a 50% hit rate.
But some of the stuff Felix has done on the financial services industry, payments, and the whole US banking structure have risen beyond "good opinion pieces" and, I think, made a real contribution to our understanding of these issues.
So I was just pointing out we have a single area where we engage because disagreement fosters debate, but I really, really liked this point and wanted to talk about his overall great work.
Joseph
Joseph:
DeleteYes, I too think Felix is great, indeed I nominated him for the American Statistical Association's Excellence in Statistical Reporting Award a few years ago.
But, in this case, if he is mistaken, I could see how your relative lack of prominence would mean that he might not spend the time and effort to get to the bottom of your argument.