Yglesias on the Dangers of Observational Data

Matt Yglesias has a piece on the Dangers of Data that really should be the Dangers of Observational Data!  True randomized or quasi-randomized experiments, when you can do them, have none of these limitations ascribed to the thermostat problem (and, in physics, an experiment is how you would figure out what the thermostat actually does). 

I am also amazed by the different foci that fields put on different methodological issues.  In observational pharmacoepidemiology we are obsessed with the issue of confounding by indication and constantly worry that it is leading to non-trivial amounts of bias.  The concept behind confounding by indication is awfully similar to the problem described by Milton Friedman's thermostat.  But I never hear economists bring that up as a major issue with observational data; perhaps because they lack experiments to tell them how often an observational estimate is wildly inaccurate (whereas in pharmacoepidemiology these experiments are slow and rare rather than non-existent). 

None of this is to say that you cannot do valid inference with observational data -- you most definitely can.  But it does highlight the need to be very, very careful.