Mark sent me this article and I thought that it made an excellent point. Research is not right or wrong depending on the source. In this sense, there is a classic use of the Ad hominem fallacy occurring where we criticize the source of the research and not the content of the research.
I think that this brings up two issues, both important.
1) Transparency: A lot of the issues about epidemiological lie in the fact that methods are not standard or transparent. There is an art to doing an epidemiological study and no good mechanism in place to determine if the results were cherry-picked or the most honest representation of the ata possible.
2) Incentives: Let us be honest, everyone in the research game is motivated (at least in part) by financial incentives. I see a lot of researchers who are also motivated by a genuine desire to help solve serious medical problems. But I do not think that being in academia is a good test for this motivation as people go into industry for a lot of reasons.
But it is, without a doubt, in the best interest of a researcher to find something "interesting". One paper can make or break the transition from post-doctoral fellow to faculty. One paper can make or break a tenure application. This is not to say that fraud is widespread in the academy -- I rather expect that it is extremely rare.
But we are kidding ourselves if we do not admit that everyone with a career in research (whether in industry, government, or the academy) doesn't have at least some incentives to find exciting and interesting results. I think we mostly resist this temptation and focus on giving the most honest appraisal of the data that is possible; but we should never forget that all research can be driven by the rewards of actual discovery.
Thursday, March 19, 2009
Wednesday, March 18, 2009
Tenure
It is no surprise that, like any junior academic, I have seriously mixed feelings about the slow erosion of tenure. I must admit that I agree with the author of Confessions of a Community College Dean that the current academic system (two tracks) is a symptom of a system in decline.
What is not addressed there, but which is really relevant to my area, is how the loss of tenure changes my job. As a researcher (without tenure and with no prospect of tenure), my career is now dependent on getting funding or working on projects that get funding. In a real sense this is the death of the "freedom to explore" that originally lured me into the academy. Now, it is quite true that there was never a utopian time where professors and post-docs could diligently pursue their whims where-ever these might lead. Certainly modern teaching loads are completely different (at least in medical schools) which really does change the character of the job.
Still, it means that my career will now be spent responding to requests for funding in a broad range of government initiatives. Long periods of funding are five years and it is quite possible to have the more appealing types of grants last two years. This is actually less stable than the contract system that the University of Kentucky is implementing!
It is not that great research can't be done under these conditions. But it does really change the definition of stability. I never thought, when I left the banking industry in 2002, that I would have less employment stability. More curious, I seem to have about the same freedom to innovate (I can innovate insofar as it advances a pre-specified set of goals).
It's certainly food for thought.
What is not addressed there, but which is really relevant to my area, is how the loss of tenure changes my job. As a researcher (without tenure and with no prospect of tenure), my career is now dependent on getting funding or working on projects that get funding. In a real sense this is the death of the "freedom to explore" that originally lured me into the academy. Now, it is quite true that there was never a utopian time where professors and post-docs could diligently pursue their whims where-ever these might lead. Certainly modern teaching loads are completely different (at least in medical schools) which really does change the character of the job.
Still, it means that my career will now be spent responding to requests for funding in a broad range of government initiatives. Long periods of funding are five years and it is quite possible to have the more appealing types of grants last two years. This is actually less stable than the contract system that the University of Kentucky is implementing!
It is not that great research can't be done under these conditions. But it does really change the definition of stability. I never thought, when I left the banking industry in 2002, that I would have less employment stability. More curious, I seem to have about the same freedom to innovate (I can innovate insofar as it advances a pre-specified set of goals).
It's certainly food for thought.
Tuesday, March 17, 2009
Pre-specificied Analysis
One thing that I always find challenging is how to handle changes in the analytic plan. If you send the first of result to the writing group to be discussed and they come back with "wouldn't it make more sense if . . ." then what do you do?
In one sense, this sort of constructive feedback can improve our understanding of an association and improve the paper. On the other hand, this rather does make the "p-values" less clear. If you pick the association with the lowest p-value are you optimizing on how to best present an association or are you picking a result that is optimized on the distribution of noise in the data.
It is pretty clear to me that with a pre-specified test of an association that you should stick the analysis plan. But what if you are exploring? Is there a rule for exploratory analysis?
In one sense, this sort of constructive feedback can improve our understanding of an association and improve the paper. On the other hand, this rather does make the "p-values" less clear. If you pick the association with the lowest p-value are you optimizing on how to best present an association or are you picking a result that is optimized on the distribution of noise in the data.
It is pretty clear to me that with a pre-specified test of an association that you should stick the analysis plan. But what if you are exploring? Is there a rule for exploratory analysis?
OT: Firefly
I was visiting a coffee shop in Seattle that was decorated with a lot of Firefly themed pictures.
So today's post is entirely about good science fiction and the thing that everyone misses. The trick to good science fiction is to start with characters and writing that would work in a standard movie without any science fiction elements at all. Then the science fiction elements can enhance the story and add to the sense of wonder and possibility.
Star Wars could have been a story of a squire becoming a knight. Star Trek could have been a sailing ship in the age of discovery. Both are enhanced by science fiction elements.
But the series that may have had the best characters was Firefly. The newer Battlestar Galactica is trying to compete but the basic story of Firefly was interesting, rich and filled with characters we liked surprisingly quickly.
It really is a shame that we'll never get to see it end.
So today's post is entirely about good science fiction and the thing that everyone misses. The trick to good science fiction is to start with characters and writing that would work in a standard movie without any science fiction elements at all. Then the science fiction elements can enhance the story and add to the sense of wonder and possibility.
Star Wars could have been a story of a squire becoming a knight. Star Trek could have been a sailing ship in the age of discovery. Both are enhanced by science fiction elements.
But the series that may have had the best characters was Firefly. The newer Battlestar Galactica is trying to compete but the basic story of Firefly was interesting, rich and filled with characters we liked surprisingly quickly.
It really is a shame that we'll never get to see it end.
Friday, March 13, 2009
Irregular Observations
One phenomena that definitely annoys me is dealing with irregular observations. This occurs in contexts were data is passively collected based on when people get medical tests. For example, blood pressure is collected when you visit a medical doctor and this information can be used to assess trends in the population.
Here is the problem: people who have no readings often come from two very distinct groups. One is composed of very healthy people who simply have no need of medical services. The second is comprised of poor compliers who should seek medical care but don't. Obviously, the trajectory of these two groups is very different. And, equally obviously, it's hard to argue that these effects will cancel out in a real population.
Inference can still be done but it makes it hard to rule out subtle issues of bias.
Here is the problem: people who have no readings often come from two very distinct groups. One is composed of very healthy people who simply have no need of medical services. The second is comprised of poor compliers who should seek medical care but don't. Obviously, the trajectory of these two groups is very different. And, equally obviously, it's hard to argue that these effects will cancel out in a real population.
Inference can still be done but it makes it hard to rule out subtle issues of bias.
Thursday, March 12, 2009
Missing Data
Is there any issue that is more persistent and more difficult to solve than missing data?
It takes a perfectly good study and layers assumptions on it. There is a clear divide in how to handle it. One option is to argue "why would you not want to use real data" and rejects the assumptions of imputation. Of course, this approach makes it's own set of strong assumptions that are often not likely to be met.
So you'd think that doing the right thing and modeling the missing data is the way to go? Well, it's an improvement but it is pretty rare that the assumptions of missing data technique are met (missing at random is just not accurate in real data).
So what do you do? Most of the time I recommend modeling (inverse probability weighting or multiple imputation) but I must confess that the lack of a solution that is actually good is rather distressing!
It takes a perfectly good study and layers assumptions on it. There is a clear divide in how to handle it. One option is to argue "why would you not want to use real data" and rejects the assumptions of imputation. Of course, this approach makes it's own set of strong assumptions that are often not likely to be met.
So you'd think that doing the right thing and modeling the missing data is the way to go? Well, it's an improvement but it is pretty rare that the assumptions of missing data technique are met (missing at random is just not accurate in real data).
So what do you do? Most of the time I recommend modeling (inverse probability weighting or multiple imputation) but I must confess that the lack of a solution that is actually good is rather distressing!
Wednesday, March 11, 2009
A few thoughts on reviews
One of the most frustrating things that we have to face as researchers are reviews. We all want people to recognize the hard work that went into developing papers and grants. None of us got to the point that we were sending material out for review except by putting in many years and a startling amount of hard work.
So it is very annoying when strange reviews come back. But I have learned that there are a few basic rules:
Should the paper be rejected:
1) Decide if the criticisms are substantive or stylistic. If substantive, then you need to either redevelop the paper or retire it. This is never a pleasant discovery but, in the long run, you'll be glad that a sharp reviewer caught an issue. In my experiences, true substantive critiques are rare.
2) If the criticism is stylistic then don't put time into it. Likely the next set of reviewers will have different preferences. Resubmit rapidly.
3) If the criticism seems to apply to another paper, entirely, then seriously consider rewriting the confusing sections for clarity. You are the subject matter expert -- it is not reasonable that reviewers and readers will necessarily follow all of the nuances.
In the same vein, as a reviewer, asking for massive redevelopment for purely stylistic reasons is often a poor choice. Ask whether the paper is methodologically valid (no point in letting mistakes into the literature) and relevant. These are the real questions that need to be considered.
Peer Review is a frustrating process but it can really improve work if you take advantage of it.
So it is very annoying when strange reviews come back. But I have learned that there are a few basic rules:
Should the paper be rejected:
1) Decide if the criticisms are substantive or stylistic. If substantive, then you need to either redevelop the paper or retire it. This is never a pleasant discovery but, in the long run, you'll be glad that a sharp reviewer caught an issue. In my experiences, true substantive critiques are rare.
2) If the criticism is stylistic then don't put time into it. Likely the next set of reviewers will have different preferences. Resubmit rapidly.
3) If the criticism seems to apply to another paper, entirely, then seriously consider rewriting the confusing sections for clarity. You are the subject matter expert -- it is not reasonable that reviewers and readers will necessarily follow all of the nuances.
In the same vein, as a reviewer, asking for massive redevelopment for purely stylistic reasons is often a poor choice. Ask whether the paper is methodologically valid (no point in letting mistakes into the literature) and relevant. These are the real questions that need to be considered.
Peer Review is a frustrating process but it can really improve work if you take advantage of it.
Tuesday, March 10, 2009
Peer Review
There is an interesting post over at Drug Monkey on the issue of bias in peer review. I think that there are two issues that really strike me as important in peer review and it is easy to confuse them. One is the idea that individual reviewers will have preferences for the types of research that they like to see done. This issue is difficult, if not impossible, to solve.
Two, and more annoying, is the issue of competence and coherence in review. I cannot enumerate the number of reviews that I have gotten that had questionable elements. I remember one journal claiming that they could not publish a paper that had an "unverifiable assumption" in it. The assumption in question, no unmeasured confounders, was a pretty standard assumption for all research. Even clinical trials have this issue with loss to follow-up not necessarily being at random.
But the reviewer is protected from strong complaints of "what were you thinking?". Now, I too have certainly done peer reviews that could have been better. I think we all can think of examples of this. So I am not claiming to be "special" and I am sure that I have been cursed by an author more than once for not "getting it".
But I think that these concerns are what gives anonymous peer review it's bad name.
Two, and more annoying, is the issue of competence and coherence in review. I cannot enumerate the number of reviews that I have gotten that had questionable elements. I remember one journal claiming that they could not publish a paper that had an "unverifiable assumption" in it. The assumption in question, no unmeasured confounders, was a pretty standard assumption for all research. Even clinical trials have this issue with loss to follow-up not necessarily being at random.
But the reviewer is protected from strong complaints of "what were you thinking?". Now, I too have certainly done peer reviews that could have been better. I think we all can think of examples of this. So I am not claiming to be "special" and I am sure that I have been cursed by an author more than once for not "getting it".
But I think that these concerns are what gives anonymous peer review it's bad name.
Monday, March 9, 2009
NIH Challenge Grants
I am beginning to think that the NIH challenge grants are a cunningly disguised trap. The NIH is giving out $200 million for grants that can be up to $1 million apiece and do not require preliminary data. This set-up could generate a lot of applications.
I think that it might make more sense to put that effort into a very solid R01 grant proposal and try to win longer term funding under less pressure of time and with more opportunity to generate productivity.
But, of course, the siren call of "no pilot data" is certainly sounding in my ears too!
I think that it might make more sense to put that effort into a very solid R01 grant proposal and try to win longer term funding under less pressure of time and with more opportunity to generate productivity.
But, of course, the siren call of "no pilot data" is certainly sounding in my ears too!
Sunday, March 8, 2009
A reply to Andrew Gelman
A reply to Andrew Gelman's latest post where he links to an old post on propensity scores:
My understanding of the issue is that there was also a prevalent user problem (creating selection bias) at least partially due to time-varying risk. While this could have been found and modeled, I am unsure about how propensity scores give any advantage over a thoughtfully constructed regression model. Unless the study you are thinking of had a lot more power to estimate predictors of exposure than outcomes due to very few outcomes (but I don't believe that this was the case with the Nurse's Health Study).
I'm not saying that better statistical models shouldn't be used but I worry about overstating the benefits of propensity score analysis. It's an extremely good technique, no question about it, and I've published on one of it's variations. But I want to be very sure that we don't miss issues of study design and bias in the process.
Issues of self-selection seriously limit all observational epidemiology. The issue is serious enough that I often wonder if we should not use observational studies to estimate medication benefits (at all). It's just too misleading.
This minor point of disagreement aside, I freely admit that Andrew Gelman is one of my heroes in the statistical community. I love some of his posts. His work on statistical significance is incredibly thought provoking, very helpful in clarifying thought and a must read for any epidemiologist.,
My understanding of the issue is that there was also a prevalent user problem (creating selection bias) at least partially due to time-varying risk. While this could have been found and modeled, I am unsure about how propensity scores give any advantage over a thoughtfully constructed regression model. Unless the study you are thinking of had a lot more power to estimate predictors of exposure than outcomes due to very few outcomes (but I don't believe that this was the case with the Nurse's Health Study).
I'm not saying that better statistical models shouldn't be used but I worry about overstating the benefits of propensity score analysis. It's an extremely good technique, no question about it, and I've published on one of it's variations. But I want to be very sure that we don't miss issues of study design and bias in the process.
Issues of self-selection seriously limit all observational epidemiology. The issue is serious enough that I often wonder if we should not use observational studies to estimate medication benefits (at all). It's just too misleading.
This minor point of disagreement aside, I freely admit that Andrew Gelman is one of my heroes in the statistical community. I love some of his posts. His work on statistical significance is incredibly thought provoking, very helpful in clarifying thought and a must read for any epidemiologist.,
Saturday, March 7, 2009
Self Selection
In a lot of ways, I think that different forms of self selection are the biggest threat to study validity in observational epidemiology. We see it in loss to follow-up when participants select out of clinical trials. We see it in important exposures like diet, exercise and alcohol use where the exposure is likely correlated with many other health seeking behaviors. Heck, we know that being adherent to placebo therapy is associated with good outcomes.
So the trick seems to isolating the effect of a single exposure. It is the process of thinking up ways to do this isolation that allows epidemiologists to really earn their keep.
So the trick seems to isolating the effect of a single exposure. It is the process of thinking up ways to do this isolation that allows epidemiologists to really earn their keep.
Friday, March 6, 2009
Why do observational epidemiology?
Observational epidemiology studies are often the source of highly misleading results. And yet, despite this problem, they are essential to the better understanding of human health. There are many exposures, some of them quite critical, that cannot be studied in any other way.
My personal favorite is adverse drug effects. Clinical trials are often underpowered to detect adverse events; in order to show these effects trials often need to be combined. Given the logistics involved, it is helpful to show an association between the drug in question and adverse events in real populations.
I hope to discuss the many interesting challenges and ideas in this field of research as I try to muddle towards some sort of resolution in a confusing sea of fuzzy data.
My personal favorite is adverse drug effects. Clinical trials are often underpowered to detect adverse events; in order to show these effects trials often need to be combined. Given the logistics involved, it is helpful to show an association between the drug in question and adverse events in real populations.
I hope to discuss the many interesting challenges and ideas in this field of research as I try to muddle towards some sort of resolution in a confusing sea of fuzzy data.
Subscribe to:
Posts (Atom)