Living with Mistakes the easy way -- larger points

Andrew Gelman fact-checks the New York Times:

For example, David Brooks wrote the following, in a column called “Living with Mistakes”:

The historian Leslie Hannah identified the ten largest American companies in 1912. None of those companies ranked in the top 100 companies by 1990.

Huh? Could that really be? I googled “ten largest american companies 1912″ and found this, from Leslie Hannah:

No big deal: two still in the top 10 rather than zero in the top 100, but Brooks’s general point still holds. As Brooks said, we have to live with mistakes. This is more a comment on how a statistician such as myself will see a number and immediately feel the urge to check it.

...

Again, this is no criticism of Brooks—as a journalist, he’s of course more interested in good stories than in getting the details right (recall the notorious $20 dinner at Red Lobster). That’s ok. Storytelling is his job, numbers are mine.

I appreciate Gelman's sharp eye and I can understand why, being a nice guy, he tends to favor catch-and-release criticism, but I disagree sharply with his conclusions here.

For starters, creative destruction a big part of the story Brooks is telling, both in this column and in his body of work collectively. In this narrative, it's a tough process, but a healthy and fundamentally fair one. Here's the sentence that precedes Gelman's excerpt from Brooks: "Even if you make it to the top, it is very hard to stay there."

We can argue about the validity of this view, but there's no question that Brooks' incorrect statement supported this point while the corrected version undercuts it. There are millions of businesses and if success is truly determined solely by who has the best business model, the best execution and the best timing, a long run in the Fortune 500 (with shifts in markets and changes in management) would be a hell of a feat. The number we actually see would certainly imply something more at work (regulatory capture, anti-competitive practices, etc.).

So yes, I would call this a big deal with respect to the story Brooks is telling.

Perhaps more importantly, I have a problem with the distinction drawn here between statisticians and the rest of the world. It's true that the ability to sniff out suspect numbers and questionable findings is an essential part of being a good statistician, but it's also part being a good journalist (and a good engineer and a good accountant and any number of other professions).

As statisticians we need strong mathematical intuition and a heightened sense of how numbers relate to each other, but we don't have any claim to the urge to check the unlikely. David Brooks was simply being a bad journalist when he wrote that passage. It was not because of inability -- Brooks is smart and highly capable -- but because he didn't care enough to get it right. He know there would be no real consequences either for him or his paper if he got it wrong.

And a lack of consequences, my friends, makes living with your mistakes amazingly easy.