Joseph recently had a post on Matt Yglesias's Slate piece on data. I pointed out in the comments that I wasn't that impressed. Here's a more detailed explanation.
I realize that Yglesias is writing for a general audience and that can muddle the issue, but I keep getting the feeling that he really doesn't get the subtlety here, that he misses the fundamental asymmetry between significance and insignificance and the importance of what Friedman and Rowe are saying.
I don't want to get caught up in the weeds of interpreting (let alone defending) p-value, but if you see a contextually significant result you pretty much know that there's a relationship there. It may not be causal or useful and it may not be what it appears, but there's something.
If you don't see a significant result, you don't know anything. Maybe there's nothing there or maybe there's too much noise or the relationship is outside the range of data or you could be using techniques that miss nonlinear relationships (an argument for CHAID and CART but that's a topic for another post). You certainly wouldn't draw a conclusion like burning fossil fuels doesn't cause heat.
As for the thermostat scenario, Yglesias focuses on this as an example of observational analysis missing a relationship. That's true but it's a very specialized case of a wide spread problem (lots of things cause us to miss relationships). Furthermore, it misses the real profundity of the analogy: it is the job of self-regulating systems to put forces in opposition, to set up destructive interference. The better the system works, the less appears to happen.
And finally, while I'm in a picky mood,
Yglesias should realize that lots of natural sciences (astronomy, meteorology) make very limited use of experiments.
"lots of natural sciences (astronomy, meteorology)" I think there is a different of kind of a passive system (astronomy, meteorology) where you can only observe (it's closed to human influence) and a dynamic system like economics where you can actually take actions. Since the actions being taken are the object of study, the simple passive observation approach isn't that helpful.
ReplyDeleteIt was the sentence " In the natural sciences what you'd do with that finding is run some experiments, and you'd figure out what was really happening." that annoyed me, but I should have been clear that the lack of experimentation did not make astronomy similar to economics. Econ and other social sciences face some unique challenges (for one thing, planets don't change their behavior in reaction to hot new theories of astrophysics).
DeleteI'm not sure we can still say that meteorology is "closed to human influence" but that's a topic for another day.
Yes, but insofar as we decide to interfere with meteors, there will be testable predictions about Newtonian mechanics involved.
Delete:-)