Bayesian Model Averaging

I know that it is the best choice for selecting variables for predictive models. I know that it is massively superior to stepwise model selection. Chris Volinsky's page is a great resource for doing these models in R.

But I don't speak R and I have been resisting for a while. It's the decade of SAS training -- I am so fast in SAS that it feels like I am walking through water when I switch to R or STATA. I can do it but the experience is unpleasant, to say the least.

On the other hand, it's the right approach for a project that I am looking at right now. So maybe it is time to bite the bullet and use a new software interface?

When non-linear relationships attack

In a lot of cases, in epidemiology, it is conventional to assume that relationships are linear. In a lot of cases this assumption seems to be pretty reasonable. Occasionally you get a relationship where very large values have biological plausibility (like income or albumin excretion rate) and a base 2 logarithmic transformation is a more logical way to proceed.

So far, so good.

Now look at the rate of rate of change of bone mineral density by age for women as reported in this article from the Canadian Medical Association Journal. Between ages 25 and 40 the rate of change of bone mineral density is positive. Then, between ages 40 and 60 the rate of change is negative. Suddenly, at age 60+, the rate of change becomes positive again. Yes, the first derivative of the relationship between age and bone mineral density is quadratic! That would rather imply that the actual relationship is cubic!!

Without seeing this sort of high quality descriptive data, I would have screamed "overfit" if I saw a cubic age term in a statistical model. As it is, I am having pessimistic thoughts about just have big the sample size needs to be in order to estimate the polynomial association between age and bone mineral density.

Now imagine that you are trying to remove confounding by age from another association; just how are you going to be sure that you don't have residual confounding?

Wow, juSt wow!

Stepwise Regression

Why is it so hard to get consensus on a good variable selection technique for exploratory models? The classic one, implemented everywhere and understood by everyone, is really sub-optimal (Stepwise regression -- I am looking at you). It seems to just love to include noise parameters which make any and all models difficult to explain.

Sure, you can build a model based on theory but what do you do when you want to know what factors might be associated with an outcome? And, of course, exploratory datasets tend to be the smallest possible data sets!!

Grrrr . . . .

Reporting results

Writing an effective scientific paper is an art form. It requires presenting a complex idea with about 3000 words. However, it is always the case that there are features of the study that are hard to describe cleanly with a small number of words. So what does one do?

If you go into detail then you inevitably confuse the reader. On the other hand, you want to produce the most complete report possible. Online supplements can help but only so far.

This happened to me in this paper:

Delaney JAC, Daskalopoulou SS, Suissa S. Traditional versus marginal structural models to estimate the effectiveness of β-blocker use on mortality after myocardial infarction. Pharmacoepidemiol Drug Saf 2009; 18(1):1-6.

There were two effects going on in the same paper. One, we were accounting for time dependent confounding. Two, we were switching from a conditional to a marginal estimate. Both of these changes contribute to the differences in estimates.

But does separating them increase or decrease confusion? If the goal is to find an analytic approach that is equivalent to a randomized controlled trial then the reasons why are less important.

My question is whether I'd aid understanding by pointing these subtle points out or if I would enhance confusion by bringing in tangential points. To this day I question which approach was correct!

Budget Cuts

One of the hardest things to handle in the modern environment is that the academy is not designed for dynamism. It is based on a system where hard work and apprenticeship eventually lead to a slow succession of improvements. However, when circumstances can change rapidly this puts the whole system into flux. Positions and outcomes that people have labored for decades to achieve can now be at risk.

I think that this situation is less true for the very senior people and more for those "too far in to back out" but who have not "really made it yet". Basically, PhD Students, post-doctoral fellows and assistant professors are the class at risk and are often placed into situation that are Kafkaesque.

Now it is hard to argue for who deserves resources; "deserve" being such a complicated word. But it is pretty clear to me that the budget cuts to higher education (at least here in the state of Washington) look to be pretty deep. I suspect few companies would cut so deeply unless they were in crisis.

This leads me to ask: is higher education failing so badly that it should count as being in crisis?

But, either way, there really is not a lot of fat, per se, left to be cut. Schools seems to be in (generally) slightly poor repair. Equipment seems to old and a lot of work-arounds exist. Teaching resources are hardly in surplus. It could be that we are trying to do too much with too little. But that suggests rethinking of priorities -- not the brutal selection of massive reductions in budgets.

Or at least that is the way that I see it.

What to do in hard times

In this time of funding cuts and stress, what is the right reaction? I think that Professor in Training nails it: keep working! Stressing about what might happen is useful only insofar as it informs contingency planning. But, as a post-doctoral fellow, I have limited options that don't involve trying to develop a record of accomplishment.

One day I might get around to telling the long story of my attempts to move forward into the academy. But, for the moment, let me say that I like this attitude. Way better than assuming that disaster is coming and seeing that as an excuse for there to be at least some excitement in life. I really hope that I never get that cynical.

I like research but I admit that I also really liked working in the private sector. It had it's downsides, but I liked the dynamism and the idea that accomplishment was to be prized. I think that is why I have liked my current unit so much -- they have the same sort of culture!

OT: Dentists

I know that sometimes you need to inflict pain to make things better in the long run. And I am a happier person with the idea that I will have some sort of teeth in middle age.

But it is not pleasant to have three hours of solid dental drilling!!

:-(

More on Tenure

There is an interesting discussion on DrugMonkey about Tenure.

I think that the original comments are making a rather important point. High value research with a short term pay-off is ideally suited to the private sector. They have every advantage in pursuing these goals and lack the "overhead" of an academic institution. I know that discussions of comparative advantage can be complicated but this situation is one where the private sectors really are better poised to solve the questions.

The advantage of the academy is in long term planning and results. This environment gives stability and the ability to pursue dead ends. Even if the academy was better at some short term goals, it's still better to have it focus on the goals where the structure is advantaged relative to the private sector.

One argument against tenure involves the complicated issue of mandatory retirement. I think that this issue is not unique to academia and it is an independent issue from tenure. It is also unclear, in a world where pensions are so unstable, what the options are. Perhaps we need to reconsider ideas like seniority based salaries? I am not sure but I see this as a more general concern and only distantly related to the issue of tenure itself.

But the real issue seems to be whether or not the post-tenure world is good for the academy. I would argue that the answer is no. Perhaps I made a very bad decision to go back into academics at this time given the current pressures but I can't help but think that the levels of deprivation seen by junior academics are dysfunctional. Young Female Scientist talks about the sorts of deprivation that junior academics undergo; after a decade of such lowered standard of living why is it seen as being "lazy or dysfunctional" to want job security?

So I think that there are many good arguments for tenure and I think many of the "anti-tenure" arguments are red herrings.

Academic Positions

Thoreau (whom I just don't read enough) has a great post on the issues with academic positions in bio-medicine. The recent doubling of the NIH budget has made it possible for the number of academics to dramatically increase. This increase led to people having very unrealistic expectations about academic jobs. In was in Physics in the 1990's when there was a contraction in the field -- I think it is fair to say that the future of Bio-Medicine is about to have some of the sam tragic outcomes.

The worst part is that I don't even have a decent alternate plan.

Cross Sectional Drug Effects

Probably the most frustrating thing in the life of a pharmacoepidemiologist is explaining why cross-sectional drug effects are impossible to estimate. People exposed to a drug at baseline have an outcome that is a composite of:

1) True Drug Effect

2) Underlying Disease Condition (indication for the drug)

It is impossible to separate these effects. So you have strange results when you analyze these data sets: such as anti-hypertensive medications often appear to increase blood pressure when you look at cross-sectional data.

This phenomenon makes it impossible to do any causal inference from a cross sectional drug study if the outcome is even remotely related to the indication. Much grief would be saved if we kept this feature of such studies in mind.

Too many names

Has anybody else noticed that the conceptual idea of the difference in rates/incidence/prevalence of a disease in the exposed and unexposed has too many definitions?

I can think of papers that use: odds ratio, relative risk, cumulative incidence ratio, prevalence ratio, rate ratio, hazard ratio . . . All of which have subtle differences.

But none of which are used consistently.

I suspect that we could do a lot of good just to clean this terminology up!

Medication discontinuation

A lot of my work has been in the area of Pharmacoepidemiology. So it was with great interest that I read the commentary in the March 15th issue of the American Journal of Epidemiology by Dr. Noel Weiss. Dr. Weiss is a brilliant epidemiologist and so it is no surprise that his commentary clearly laid out the conceptual and practical problems associated with these studies.

The main problem is that people do not start (or stop) medication at random. They take medications to treat some underlying condition (thus leading to confounding by indication) and they stop for a number of reasons (including the treatment is completed). We know, for sure, that some drugs have withdrawal issues (consider morphine or SSRIs).

I've actually looked at this question with statin drug withdrawal and still worry about how successful we were at controlling for confounding factors (and, believe me, we did an enormous amount of analysis to see how robust the effect was).

But what is hard, in all of these studies. is separating the reason for stopping the drug from the outcome. If a participant stops an SSRI and has an increased risk of suicide is that a marker of:

1) The drug was not working to begin with

2) There were severe withdrawal issues

Separating these two factors is important! After all, if there is a period of increased danger than alternative monitoring for serious health events becomes an option.

But Dr. Weiss gives an excellent summary of all of the things that can go wrong in such analyses and why we need to be careful in interpreting them. So if you work in drug research at all, this article is definitely worth a look.

Mistakes

We all make mistakes. Perfection, much as we would like it to be a standard part of the human condition, is not something that any one of us can claim. But, when doing science, it can happen and it is one of the hardest things to do to admit to a mistake. In all of the cases that have happened with me, it occurs during the sanity checking phase of an analysis.

But the problem is that finding issues with the analysis (i.e. did we define a quantity correctly) unnerves collaborators. Rather than being happy that we are having a good "back and forth discussion" and being pleased that the issues have come up early, it seems to shake confidence.

I think that this tendency to react badly to these issues is actually harmful. Mostly because it makes analysts less likely to speak up if they suspect that something is wrong. And, should I ever become an independent investigator, I really want people to speak up if there is an issue with data or analysis.

So I am not sure how to structure these interactions to provide the best possible outcome!

Service work

The more epidemiology that I do, the less I like being an analyst. It is a very annoying job. People expect you to do data management, analysis and a fair chunk of the writing (methods section!) that is often the least interesting to craft. But there is a common tendency to do two things:

1) Make all decisions without talking to the analyst

2) Act like it is your fault if the data isn't ideal

I used to volunteer for these jobs because I thought that it would be fun and interesting to work with many different people. Now I must admit I am coming to loathe them!

Academic positions

Thoreau has an interesting article about the perverse effects that can happen when trying to encourage diversity. What I am beginning to realize is that the academic career path is a high risk/moderate reward path. Some students will end up with decent jobs that give freedom and opportunities to think; others will end up in quite different areas than they intended. But the standard academic program has a lot of the characteristics of a pyramid scheme in that the goal that is placed before PhD students, tenured faculty positions, is increasingly unrealistic.

In epidemiology we are seeing an increasing reliance on soft money positions. I am not sure, in the long run, whether this reliance is a good or a bad feature. But it sure makes the implicit bargain of "years of sacrifice" being rewarded less obvious.

But the real issue with faculty seems to be the training versus placement issues. Most of the other concerns are perverse effects of that particular "elephant in the room".