Friday, August 10, 2012

Yglesias on the Dangers of Observational Data

Matt Yglesias has a piece on the Dangers of Data that really should be the Dangers of Observational Data!  True randomized or quasi-randomized experiments, when you can do them, have none of these limitations ascribed to the thermostat problem (and, in physics, an experiment is how you would figure out what the thermostat actually does). 

I am also amazed by the different foci that fields put on different methodological issues.  In observational pharmacoepidemiology we are obsessed with the issue of confounding by indication and constantly worry that it is leading to non-trivial amounts of bias.  The concept behind confounding by indication is awfully similar to the problem described by Milton Friedman's thermostat.  But I never hear economists bring that up as a major issue with observational data; perhaps because they lack experiments to tell them how often an observational estimate is wildly inaccurate (whereas in pharmacoepidemiology these experiments are slow and rare rather than non-existent). 

None of this is to say that you cannot do valid inference with observational data -- you most definitely can.  But it does highlight the need to be very, very careful. 


  1. Hate to be a broken record, but I'm just not that impressed with this, particularly when compared with Nick Rowe's treatment

  2. I agree that the Nick Rowe piece was the better piece of work, by far. I was making a fast point and picking the simpler article.

    By the way, when did you become the one to defend excellence in Canadian bloggers? :-)