A really nice article by Andrew Gelman and Kaiser Fung

Andrew Gelman and Kaiser Fung have an article on Freakonomics in American Scientist. My favorite part was the story of Emily Oster and her theory of Hepatitis B:

Monica Das Gupta is a World Bank researcher who, along with others in her field, has attributed the abnormally high ratio of boy-to-girl births in Asian countries to a preference for sons, which manifests in selective abortion and, possibly, infanticide. As a graduate student in economics, Emily Oster (now a professor at the University of Chicago) attacked this conventional wisdom. In an essay in Slate, Dubner and Levitt praised Oster and her study, which was published in the Journal of Political Economy during Levitt’s tenure as editor:

[Oster] measured the incidence of hepatitis B in the populations of China, India, Pakistan, Egypt, Bangladesh, and other countries where mothers gave birth to an unnaturally high number of boys. Sure enough, the regions with the most hepatitis B were the regions with the most “missing” women. Except the women weren’t really missing at all, for they had never been born.

Oster’s work stirred debate for a few years in the epidemiological literature, but eventually she admitted that the subject-matter experts had been right all along. One of Das Gupta’s many convincing counterpoints was a graph showing that in Taiwan, the ratio of boys to girls was near the natural rate for first and second babies (106:100) but not for third babies (112:100); this pattern held up with or without hepatitis B. In a follow-up blog post, Levitt applauded Oster for bravery in admitting her mistake, but he never credited Das Gupta for her superior work. Our point is not that Das Gupta had to be right and Oster wrong, but that Levitt and Dubner, in their celebration of economics and economists, suspended their critical thinking.

I think that this story actually has two elements. One is the dangers of a convincing explanation. There are a lot of associations that can appear and would be extremely exciting if they were true. Just consider the recent article on statins reducing mortality due to pneumonia: it is an amazing result that would be extremely exciting if it were true. I worry that these kinds of exciting results get a lot of press instead of being seen a signposts towards needing to examine the problem more carefully. After all, it was a good thing that Das Gupta had a chance to look at her data and control for an additional predictive variable. What is concerning is not raising the idea -- it is the strength of the language: "Except the women weren’t really missing at all, for they had never been born" which implied a lot more certainty than seemed warranted. But putting that point aside, the real interesting thing (to me) is considering likely effect sizes. When you look at the population level infection rates (incremental on the infection rates in countries without this gender imbalance) then you quickly conclude that the effect of infection has to be high. After all, the rate in India appears to be about 3% (versus less than 1% in the United States). At the same time, the sex ratio in India was 1.10 (these are approximate numbers). So if the natural sex ratio is 105 and India has 110 we can do a calculation. Assume that the Hep B rate among reproductive age women is triple the population average (say 9%). So 0.91 x 105 + 0.09 x [RATE] = 110. That suggests that the sex ratio among infected women is 160 (it gets a lot worse if you merely assume double). That means we could prove this hypothesis by following a very small cohort of Hep B infected pregnant women, since the effect size is so large. Now this is a simplistic way to look at the problem, and I am sure that more nuanced approaches make sense. But isn't this the sort of data you'd look for before suggesting that the experts completely missed the explanatory variable? After all, you are positing an enormous effect size for the influence of the virus on sex ratios. This would be observed in routine clinical practice. So, not to give anybody a hard time. We all have challenges in our research and it is really hard to tackle these types of problems. People should have credit for putting their necks out and proposing testable hypotheses that can enhance our understanding of the world. But I think we should rethink just how certain we are when we make these proposals. Maybe we need to learn to say "this is a possible explanation for some of the observed variation".