Another cautionary tale of corruption by metrics

This American Life has a great piece of radio up on their site. (You can get a free download if you hurry.) It's the account of a patrol man who started collecting evidence of rule-breaking in the department. The story itself is remarkable but what makes this piece extraordinary are the audio segments that the policeman covertly recorded, including the incredibly suspenseful confrontation near the end.

There are any number of interesting angles to this story but the one that caught my ear (at least with respect to the discussions we've been having here) is the role that metrics played.

Ira Glass

The New York Police Department declined our request to come onto the radio or to have the officers who supervised Adrian Schoolcraft, and who are heard on his recordings, to be interviewed about their side of all this. But the pressure on police commanders to get better numbers really goes back to 1994, when New York started tracking crimes with a system called CompStat. CompStat, for the first time, gave commanders timely, accurate data once a week on what crimes are happening, so they could send more cops to deal with it. Chances are you've heard of all this. It became one of the best known successes in modern policing. Serious crime has dropped an astonishing 77% in New York City since CompStat began in 1994. Other cities very quickly started imitating it-- DC, Philly, LA. Baltimore's version of CompStat ended up in a recurring plot line on the TV show The Wire, where street cops are told by the bosses to do anything to pump up their numbers. And the problem with CompStat, says Professor Eli Silverman, who studies the way police forces use numbers, is that the early success of CompStat created the expectation that numbers must get better every single year, no matter what.

Eli Silverman
In the beginning it was like an orange. You could squeeze juice from an orange in the beginning much more readily than you can as you extract juice from that orange. And now, it gets harder and harder to drive crime down, because you're compared to not how you were in '94, but how you were last year the same week. And when something's pushed to the excess that it is now, and numbers dominate the system, that's when you have negative consequences.

Specifically consequences like this:

Ira Glass
Then when the officer tried to file it that way, because he didn't have a name for the unauthorized driver, he couldn't file it at all. So the robbery went unreported. Rules go into effect in the 81st precinct that make it harder to report serious crimes. Officers are told that if there's a robbery, one of their supervisors has to come out to the scene themselves. And robbery victims are told that if they don't come into the police station, no crime report will be filed at all. After Graham Rayman started publishing these stories about Adrian Schoolcraft, retired cops and some on-duty cops started contacting him with their own anecdotes about crimes being downgraded from serious to much less serious-- the most shocking of these from a high ranking detective name Harold Hernandez.

Graham Rayman
He's a very distinguished detective. He was working in the 33rd precinct in Washington Heights. And one morning he comes into work and there's a guy who's accused of first degree rape sitting in his interview room. So he sits down and he looks at the guy. And he has a little twinge, and he says, have you ever done this before? And the guy said, yeah. And Hernandez says, how many times? And he says, oh, I don't know, seven or eight. And Hernandez says, where? And he goes, in this neighborhood. And Hernandez is now dumbstruck because there's been no report of a serial rapist-- sexual predator-- working the neighborhood.

Ira Glass
Like, no crimes have shown up. People haven't shown up saying they've been raped or assaulted.

Graham Rayman
He hasn't been notified. And he would be notified as a senior detective in the unit. It would be a very big deal. And so he says, can you give me the dates and locations? And the guy says, well, I can try, but you're going to have to take me around and I'll show. I'll show you. So he and a fellow detective get in the car and they drive around. And they look, and the suspect-- whose name is Darryl Thomas-- points out the locations. And then Hernandez takes his notebook and he writes down the locations. And then he goes back and he looks through stacks of crime complaints. And he finds them. And he realizes that they've been classified-- they've been downgraded. They've been classified either as criminal trespassing or criminal possession of a weapon-- both relatively minor crimes, given that the actual conduct in the narrative that the victims are describing is either first degree burglary, robbery, or sexual abuse, sexual assault. And he confronts his bosses about it. He confronts the precinct commander. And he confronts his detective squad commander. And everyone just shrugs. Meanwhile everyone's terrified that it's going to come out-- that these women are going to go to the press, and it's going to be a huge embarrassment, a huge scandal for the department. And if it had come out, it would have been a huge scandal for the department. But the department was able to keep it quiet. The District Attorney's office prosecuted Thomas and he went away for 50 years. But here's the interesting part-- they never publicized the case. There was never a press release issued about it. There was never a news article written about the case.

One of the reasons that metric-centric systems can become so bad is that they almost always start out as something good. Compstat, like standardized testing, was a good thing and still is, if used properly. At some point though, people start to focus more on the metric, not because increased focus is helpful but because a metric-based approach is so seductively simple. When that happens the metric becomes an end rather than means (the map becomes the thing for all you Korzybski fans in the audience).

That's generally when things go to hell.