Levitt and publishing bias in medical journals

Via Andrew Gelman here is a quote from Steven Levitt

When I told my father that I was sending my work saying car seats are not that effective to medical journals, he laughed and said they would never publish it because of the result, no matter how well done the analysis was. (As is so often the case, he was right, and I eventually published it in an economics journal.)

Now compare his article to this one (published a year later):

OBJECTIVE: The objective of this study was to provide an updated estimate of the effectiveness of belt-positioning booster (BPB) seats compared with seat belts alone in reducing the risk for injury for children aged 4 to 8 years. METHODS: Data were collected from a longitudinal study of children who were involved in crashes in 16 states and the District of Columbia from December 1, 1998, to November 30, 2007, with data collected via insurance claims records and a validated telephone survey. The study sample included children who were aged 4 to 8 years, seated in the rear rows of the vehicle, and restrained by either a seat belt or a BPB seat. Multivariable logistic regression was used to determine the odds of injury for those in BPB seats versus those in seat belts. Effects of crash direction and booster seat type were also explored. RESULTS: Complete interview data were obtained on 7151 children in 6591 crashes representing an estimated 120646 children in 116503 crashes in the study population. The adjusted relative risk for injury to children in BPB seats compared with those in seat belts was 0.55. CONCLUSIONS: This study reconfirms previous reports that BPB seats reduce the risk for injury in children aged 4 through 8 years. On the basis of these analyses, parents, pediatricians, and health educators should continue to recommend as best practice the use of BPB seats once a child outgrows a harness-based child restraint until he or she is at least 8 years of age.

 So what is different?  Well, the complete interview data is a hint as to what could be happening differently.  It is very hard to publish a paper in medical journal using weaker data than that present elsewhere.  Even more interestingly, papers before this one found protective associations (this was 2006) which should also be concerning. 

Then we notice that the Doyle and Levitt has Elliott et al. as a reference, but still claim that they are the first to consider this issue:

This study provides the first analysis of the relative effectiveness of seat belts and child safety seats in preventing injury based on representative samples of police-reported crash data.

So now let us consider reasons that a medical journal may have had issues with this paper.  First, it does not seem to deal with the previous literature well.  Second, it doesn't explain why crash testing results do not seem to translate into actual reduction in events.  It might be due to misuse of the equipment, but it is not clear to me what the conclusion should be then. 

But it seems that jumping to the conclusion that the paper would not be published because of the conclusion seems to assume facts not in evidence.  It is common for people to jump fields and apply the tools that they have learned in their discipline (economics) and not necessarily think about the issues that obsess people in the field (public health).  Some times this can be a good thing and a new perspective can be a breath of fresh air.  But in a mature field it can also be the case that there is a good reason that the current researchers focus on the points that they do.

This reminds me of Emily Oster, another economist who wandered into public health and seemed surprised at the resistance than she encountered.

So is the explanation Levitt's father gave possible?  Yes.  But far more likely was the difficulty of jumping into a field with a high counter-intuitive claim and hoping for an immediate high impact publication.  Medical journals are used to seeing experiments (randomized controlled drug trials, for example) overturn otherwise compelling observational data.  So it isn't a mystery why the paper had trouble with reviewers and it does not require any conspiracy theories about public health researchers not being open to new ideas or to data. 

Freakonomics

Andrew Gelman's blog has a nice discussion of Freakonomics that is very topical given the discussion of Mike Daisy.  I think that he was a pretty balanced response to Stephen Dubner,who seemed to be rather distressed by the Andrew Gelman and Kaiser Fung response. Instead, I think that pointing out issues in a provocative and thought-provoking blog is essential. I admit that I often get very frustrated with the constant criticism of peer review. But it is essential to have errors pointed out and I have not seen a better way to have that happen then to have the mistakes repeatedly pointed out -- it sure makes me more careful as an epidemiologist.

One piece that I do think is worth reflecting on is this one:

Their first example of a “mistake” concerns a May, 2005, Slate column we wrote about the economist Emily Oster’s research on the “missing women” phenomenon in Asia. Her paper, “Hepatitis B and the Case of the Missing Women,” was about to be published in the Aug. 2005 issue of the Journal of Political Economy. At the time, Levitt was the editor of JPE, and Oster’s paper had been favorably peer-reviewed.  

Oster argued that women with Hepatitis B tend to give birth to many more boys than girls; therefore, a significant number of the approximately 100 million missing females might have been lost due to this virus rather than the previously argued explanations that included female infanticide and sex-selective mistreatment.  

Other scholars, however, countered that Oster’s conclusion was faulty. Indeed, it turned out they were right, and she was wrong. Oster did what an academic (or anyone) should do when presented with a possible error: she investigated, considered the new evidence, and corrected her earlier argument. Her follow-up paper was called “Hepatitis B Does Not Explain Male-Biased Sex Ratios in China.”

 I think that this missed the point of what was causing concern with this article.  An economist wanders into public health and overturns the conventional wisdom completely by considering a possible predictor but not really understanding why epidemiologists had not considered a disease-based explanation before.  It should not be considered a small point that the article showed up in an economics journal and not in a journal where it would be reviewed by experts in the clinical area. 

Is this necessary wrong to have reported potentially exciting new results?  No.  It is also true that people did put the effort into reporting when the understanding changed.  But this was in a well developed area of public health with very high policy stakes and people willing to put in a lot of effort to understand if there could be an alternate explanation.  So it induces some skepticism about "counter-intuitive" claims in areas where there are not the resources to scrutinize these claims deeply. 

Now it is natural that research has an error rate.  I wish it did not (especially not my research).  But it does point out the hazards of popularizing prelimary results.  I think I am especially sensitive to this issue as no field is more guilty of alarming and counter-intuitive findings than pharmacoepidemiology.  So I look for clues that make me cautious about publicizing preliminary results before they are really ready for prime time. 

Interesting Trading Idea

The reformed broker has a great post on ten things not to do as an individual investor. Here is one that I think is worth keeping in mind in a more general sense:

"The market is all betting one way so of course I'm betting the other way." This works very well at major turning points, which are very rare. In truth, the herd usually outsmarts the remnant and you're much better off being an ordinary zebra in the middle of the pack than straying off on your own into a deep ravine where predators lurk. If there is a turning point you see coming and you want to exploit it, fine - just don't bet your life on it and deploy all your capital at once. Also, keep in mind that not everything mean-reverts, some things simply trend - some investments will simply never come back to where you wish to buy or sell them regardless of historic price points that might make sense to you. And being contrarian just for the sake of being different is not the same as being contrarian because you see something that others don't.

You see this tendency in a lot of fields. It is psychologically fulfilling to see oneself as the loner, who can see the truth that the rest of the herd so blindly overlooks. Unfortunately, there can be a very good reason that the herd is stampeding away from the vicious lion!

I've personally watched people lose money with the notion of mean-reversion (do we still think that AOL will revert back to its tech bubble peak?). I think we see the same phenomenon in academic research programs. It seems so cool to find this interesting and counter-intuitive finding that nobody else has seen. Because they are idiots. Or, perhaps (just perhaps), because I've managed to overlook something crucial to the whole process.

So if the finding seems to explain a complex phenomenon in a simple way . . . beware. There might be a reason that the herd is off chasing the complex solution and ignoring the counter-intuitive position.

Note to Gelman -- first fill its mouth with salt, then light candles, then decapitate

Andrew Gelman is once again going after the voting-is-irrational zombie (disinterred this time by the Freakonomics team). Gelman shows, using estimates that if anything err on the conservative side, that the possibility of influencing an election, though small, can still easily be associated with a reasonable expected value.

This particular zombie has been shambling through the dark corridors of pop econ books and columns for years now (Gelman himself has been monster hunting since at least 2005), but every time the creature seems truly dead and buried, along comes someone like Landsburg or Levitt, someone who's smart enough and mathematically literate enough to know better, but who just can't resist digging up the grave.

More questions about the statistics of Freakonomics

Felix Salmon is on the case:

There’s a nice empirical post-script to the debate over the economic effects of classifying the Spotted Owl as an endangered species. Freakonomics cites a study putting the effect at $46 billion, but others, including John Berry, who wrote a story on the subject for the Washington Post, think it’s much closer to zero.

And now it seems the Berry side of the argument has some good Freakonomics-style panel OLS regression analysis of the microeconomy of the Pacific Northwest to back up its side of the argument. A new paper by Annabel Kirschner finds that unemployment in the region didn’t go up when the timber industry improved, and it didn’t go down when the timber industry declined — not after you adjust for much more obvious things like the presence of minorities in the area.

Meta-Freakonomics

Joseph recently wrote a post referring to this post by Andrew Gelman (which was based on a series of posts by Kaiser Fung which check the veracity of various claims in Superfreakonomics -- welcome to the convoluted world of the blogosphere). Joseph uses Dr. Gelman's comments about the poor editing and fact-checking of the book to make a point about the disparity between the contribution editing makes and how little we reward it. He ought to know; I have frequently taken advantage of his good nature in this area, but at the risk of being ungrateful, I don't think the point applies here. Rather than being helpful, the kind of criticism Joseph and Gelman describe could only hurt Superfreakonomics.

Or put another way, if we approach this using the techniques and assumptions of the Freakonomics books, we can show that by foregoing a rigorous internal review process the authors were simply acting rationally.

Before we get to the actual argument, we need to address one more point in Joseph's post. Joseph says that providing critical read "is one of the most helpful things a colleague can do for you, yet one of the least rewarded." This statement is absolutely true for easily 99.9% of the books and manuscripts out there. It is not, however, true for the Freakonomics books. Between their prestige and the deep pockets of William Morrow, Levitt and Dubner could have gotten as many highly-qualified internal reviewers as they wanted, reviewers who would have been compensated with both an acknowledgment and a nice check. (Hell, they might even get to be in the movie.)

But if the cost and difficulty of putting together an all-star team of reviewers for Superfreakonomics would have been negligible, how about the benefits? Consider the example of its highly successful predecessor. Freakonomics was so badly vetted that two sections (including the book's centerpiece on abortion) were debunked almost immediately. The source material for the KKK section was so flawed that even Levitt and Dubner disavowed it.

These flaws could have been caught and addressed in the editing process but how would making those corrections help the authors? Do we have any reason to believe that questionable facts and sloppy reasoning cost Levitt and Dubner significant book sales (the book sold over four million copies)? That they endangered the authors' spot with the New York Times? Reduced in any way the pervasive influence the book holds over the next generation of economists? Where would Levitt and Dubner have benefited from a series of tough internal reviews?

Against these elusive benefits we have a number of not-so-hard-to-find costs. While the time and money required to spot flaws is relatively minor, the effort required to address those flaws can be substantial.

Let's look at some specifics. Kaiser Fung raises a number of questions about the statistics in the "sex" chapter (the one about female longevity is particularly damning) and I'm sure he overlooked some -- not because there was anything wrong with his critique but because finding and interpreting reliable data on a century of sex and prostitution is extraordinarily difficult. It involves measurement covert behavior that can be affected by zoning, police procedures, city politics, shifts in organized crime,and countless other factors. Furthermore these same factors can bias the collection of data in nasty and unpredictable ways.

Even if all of the sex chapter's underlying economics arguments were sound (which they are, as far as I know), there would still have been a very good chance that some reviewer might have pointed out flawed data, discredited studies, or turned up findings from more credible sources that undercut the main hypotheses. That doesn't mean that the chapter couldn't be saved -- a good team of researchers with enough time could probably find solid data to support the arguments (assuming, once again, that they were sound) but the final result would be a chapter that would look about the same to the vast majority of readers and external reviewers -- all cost, no benefit.

Worse yet, think about the section on the relative dangers of drunken driving vs. drunken walking. These cute little counter-intuitive analyses are the signature pieces of Levitt and Dubner (and were associated with Dr. Levitt before he formed the team). They are the foundation of the brand. Unfortunately, counter-intuitive analyses tend to be fragile creatures that don't fare that well under scrutiny (intuition has a pretty good track record).

The analysis of modes of drunken transportation would be one of the more fragile ones. Most competent internal reviewers would have had the same reaction that Ezra Klein had:

You can go on and on in this vein. It's terrifically shoddy statistical work. You'd get dinged for this in a college class. But it's in a book written by a celebrated economist and a leading journalist. Moreover, the topic isn't whether people prefer chocolate or vanilla, but whether people should drive drunk. It is shoddy statistical work, in other words, that allows people to conclude that respected authorities believe it is safer for them to drive home drunk than walk home drunk. It's shoddy statistical work that could literally kill somebody. That makes it more than bad statistics. It makes it irresponsible.

Let me be clear. I am not saying that Levitt and Dubner knew there were mistakes here. Quite the opposite. I'm saying they had a highly saleable manuscript ready to go which contained no errors that they knew of, and that any additional checking of the facts, the analyses or logic in the manuscript could only serve to make the book less saleable, to delay its publication or to put the authors in the ugly position of publishing something they knew to be wrong.

Gelman closes his post with this:

It's the nature of interesting-but-true facts that they're most interesting if true, and even more interesting if they're convincingly true.

Perhaps, but Levitt and Dubner have about four million reasons that say he's wrong.