Joseph recently had a post on Matt Yglesias's Slate piece on data. I pointed out in the comments that I wasn't that impressed. Here's a more detailed explanation.
I realize that Yglesias is writing for a general audience and that can muddle the issue, but I keep getting the feeling that he really doesn't get the subtlety here, that he misses the fundamental asymmetry between significance and insignificance and the importance of what Friedman and Rowe are saying.
I don't want to get caught up in the weeds of interpreting (let alone defending) p-value, but if you see a contextually significant result you pretty much know that there's a relationship there. It may not be causal or useful and it may not be what it appears, but there's something.
If you don't see a significant result, you don't know anything. Maybe there's nothing there or maybe there's too much noise or the relationship is outside the range of data or you could be using techniques that miss nonlinear relationships (an argument for CHAID and CART but that's a topic for another post). You certainly wouldn't draw a conclusion like burning fossil fuels doesn't cause heat.
As for the thermostat scenario, Yglesias focuses on this as an example of observational analysis missing a relationship. That's true but it's a very specialized case of a wide spread problem (lots of things cause us to miss relationships). Furthermore, it misses the real profundity of the analogy: it is the job of self-regulating systems to put forces in opposition, to set up destructive interference. The better the system works, the less appears to happen.
And finally, while I'm in a picky mood,
Yglesias should realize that lots of natural sciences (astronomy, meteorology) make very limited use of experiments.
Official Information and Open Data
41 minutes ago