Someone arguing in Levitt's defence might say "well, no one could have known that Oster's hypothesis would turn out to be wrong." Could they? In 2005, the year that Oster's paper appeared in the JPE, Monica Das Gupta published a rebuttal in the Population and Development Review. She describes the results of a 1993 paper by Zeng et al, one cited by Oster:...the sex ratio at birth varies sharply by the sex composition of the living children the woman already has.... Zeng et al. show that the sex ratio at birth was normal (1.056) for first births. For second births, it was strikingly different depending on whether the first child was male or female: women whose first child was a son had a low sex ratio (1.014) for the second child, while those whose first child was a daughter had a very high sex ratio (1.494) for the second child.
To produce a pattern like that, Hep B has to be one heck of a smart virus. So the first point is: anyone with even a passing familiarity with the literature would know there was something suspicious about the Oster results.
This is actually a really good point and one that deserves more thought. The hypothesis being put forward had very little chance of being true given the actual literature cited and yet it was widely accepted as an important theory (being given wide publicity). Why is this?
I think that modern academics love the counter-intuitive theory that turns conventional thinking on its head. These are compelling stories because they seem to show how careful observation and being clever can reveal important secrets. But the very fact that these theories rely on clever stories and unexpected twists makes them more likely (and not less likely) to be incorrect.
In a sense, this feature is what I dislike about instrumental variables. One needs to tell a story about why an instrument actually has the correct statistical properties. But this relies on strong and unverifiable assumptions that cannot be directly tested. So one ends up telling an interesting story . . . but it is one that could well be wrong.
Dr Oster is a very good scientist and I don't want to generalize to the rest of her work. But it is a trap we should all look out for!
Yup. Das Gupta also had some excellent graphs:
ReplyDeletehttp://www.stat.columbia.edu/~cook/movabletype/archives/2007/05/chinas_missing.html
I don't know what's up with the Freakonomics thing, but Levitt does seem to have difficulty admitting that he's made a mistake.
Nice graphs! Figure 1 was especially compelling but the time chart was interesting as well.
ReplyDeleteI think Levitt's schtick is about being insightful; this would be undermined if many of the cool observations he makes weren't true. On the other hand, I prefer these theories at the level of interesting conversation as looking at problems in odd ways can overturn conventional wisdom. It is just hard to do!