The unspoken rule is that at least 50% of the studies published even in top tier academic journals – Science, Nature, Cell, PNAS, etc… – can’t be repeated with the same conclusions by an industrial lab. In particular, key animal models often don’t reproduce. This 50% failure rate isn’t a data free assertion: it’s backed up by dozens of experienced R&D professionals who’ve participated in the (re)testing of academic findings.
Of course, I worry even more about softer disciplines where the difficulties of replication are much higher (you don;t just have to replicate a lab, you may need to develop an entire new cohort study).
Scary stuff.
While the initial investigators may be biased to getting a result, the re-testers may be biased to *not* getting a result. It's a huge waste of their time if they keep getting identical results, the job is only interesting if their work proves something different. Therefore, how much of the 50% is due to the original investigators and how much is due to the re-testers.
ReplyDeleteGood point. That 50% may not be a perfect estimate but the idea that major journal articles are not replicating is concerning. The insistence on replication studies is one thing that the genetic epidemiologists have right.
ReplyDeleteI don't agree with the 'softer disciplines' tag. I am convinced that just as good replication (or falsification) is possible in the social sciences. It's entirely due to the lack of funding for this sort of thing. I know I had a great replication study lined up and couldn't find anyone interested in funding it. And humanities and social sciences need replication just as much as science.
ReplyDeleteThis 50% just shows how bogus this "science by step by step small contributions and testing of claims" story is. I wonder how much stuff commonly distributed in science textbooks is actually not properly tested and just passed down by consensus and tradition.
@mpledger If you read the original post, the testers are industry scientists who want to base new drugs on published research. Therefore their bias should be towards successful replication. But of course, they may be just as error prone as the original researchers.
ReplyDelete