Can we do observational medical research?

Andrew Gelman has a really nice post on observational medical research.  How could I not respond?

In the post he quotes David Madigan who has a fairly strong opinion on the matter:

I’ve been involved in a large-scale drug safety signal detection project for the last two or three years (http://omop.fnih.org). We have shown empirically that for any given safety issue, by judicious choice of observational database (we looked at 10 big ones), method (we looked at about a dozen), and method setup, you can get *any* answer you want – big positive and highly significant RR or big negative and highly significant RR and everything in between. Generally I don’t think there is any way to say definitively that any one of these analysis is a priori obviously stupid (although “experts” will happily concoct an attack on any approach that does not produce the result they like!). The medical journals are full of conflicting analyses and I’ve come to the belief that, at least in the medical arena, the idea human experts *know* the *right* analysis for a particular estimand is false.

This seems overly harsh to me.  Dr. Madigan (who I think is an amazing statistician) is working with OMAP, which I recall as being comprised of data sets of fairly low quality data (prescriptions claims for Medicare/MedicAid, GPRD and other clinical databases, and these sorts of databases).  It is a necessary evil to get the power to detect rare (but serious) adverse drug outcomes.  But these databases are often problematic when extended beyond extremely clear signal detection issues.  

The clearest example of high quality medical data is likely to be randomized controlled double-blinded clinical trials.  But there is a whole layer of data between these two extremes of data quality (prospective cohort studies, for example) that has also generated a lot of important findings in medicine.

Sure, it is true that the prospective cohort studies tend to be underpowered to detect rare adverse drug side effects (for precisely the same reason that RCTs are).  But there is a lot of interesting observational medical research that does not generate conflicting results or where the experts really seem to have a good grasp on the problem.  The links between serum cholesterol levels and cardiovascular events, for example, seems relatively solid and widely replicated.  So do the links between smoking and lung cancer (or cardiovascular disease) in North American and European populations.  There is a lot that we can learn with observational work.

So I would be careful to generalize to all of medical research.

That being said, I have a great deal of frustration with medical database research for a lot of the same reasons as David Madigan does.  I think the issues with trying to do research in medical claims data would be an excellent series of posts as the topic is way too broad for a single post.

Antidepressants

It seems like this drug class is getting more popular, with 10% of Americans taking these medications. So I go to Pub Med and notice:

I take a look at the number needed to harm (see: Arroll B, Elley CR, Fishman T, Goodyear-Smith FA, Kenealy T, Blashki G, Kerse N, Macgillivray S. Antidepressants versus placebo for depression in primary care. Cochrane Database Syst Rev. 2009 Jul 8;(3):CD007954.) and quickly conclude that either 10% of Americans are depressed (which is a silent and important epidemic) or else there are a lot people having unexpected adverse drug effects due to overtreatment.

I do know that my work in population cohorts suggests that 10% of people having clinically significant levels of depressive symptoms is a very high estimate.

Evidence

John D Cook has a very nice post up about evidence in science:

Though it is not proof, absence of evidence is unusually strong evidence due to subtle statistical result. Compare the following two scenarios.

Scenario 1: You’ve sequenced the DNA of a large number prostate tumors and found that not one had a particular genetic mutation. How confident can you be that prostate tumors never have this mutation?

Scenario 2: You’ve found that 40% of prostate tumors in your sample have a particular mutation. How confident can you be that 40% of all prostate tumors have this mutation?

It turns out you can have more confidence in the first scenario than the second. If you’ve tested N subjects and not found the mutation, the length of your confidence interval around zero is proportional to N. But if you’ve tested N subjects and found the mutation in 40% of subjects, the length of your confidence interval around 0.40 is proportional to √N. So, for example, if N = 10,000 then the former interval has length on the order of 1/10,000 while the latter interval has length on the order of 1/100. This is known as the rule of three. You can find both a frequentist and a Bayesian justification of the rule here.

Absence of evidence is unusually strong evidence that something is at least rare, though it’s not proof. Sometimes you catch a coelacanth.

Now it is true that this approach can be carried too far. The comments section has a really good discussion of the limitations of this type of reasoning (it doesn't handle sudden change points well, for example).

But it is worth noting that a failure to find evidence (despite one's best attempts) does tell you something about the distribution. So, for example, the failure to find a strong benefit for users of Vitamin C on mortality, despite a number of large randomized trials, makes the idea that this drug is actually helpful somewhat less likely. It is true we could look in just one more population and find an important effect. Or that it is only useful in certain physiological states (like the process of getting a cold) which are hard to capture in a population based study.

But failing to find evidence of the association isn't bad evidence, in and of itself that the association is unlikely.

P.S. For those who can't read the journal article, the association between Vitamin C and mortality is Relative Risk 0.97 (95% Confidence Interval:0.88-1.06), N=70 456 participants (this includes all of the trials).