More on Tyler Cowen and IHME

This is Joseph

EDIT: Andrew Gelman is also on the case with the IHME model. In case the stuff below isn't clear, critiquing this model is not only healthy but essential for good science. 

Tyler Cown writes:

And here is a further paper on the IMHE model, by statisticians from CTDS, Northwestern University and the University of Texas, excerpt from the opener:
In excess of 70% of US states had actual death rates falling outside the 95% prediction interval for that state, (see Figure 1)
The ability of the model to make accurate predictions decreases with increasing amount of data. (figure 2)
Again, I am very happy to present counter evidence to these arguments.  I readily admit this is outside my area of expertise, but I have read through the paper and it is not much more than a few pages of recording numbers and comparing them to the actual outcomes (you will note the model predicts New York fairly well, and thus the predictions are of a “train wreck” nature).

This is all fair and part of what the Epidemiology community has been discussing. But the follow-up is:

So now really is the time to be asking tough questions about epidemiology, and yes, epidemiologists.  I would very gladly publish and “signal boost” the best positive response possible. 

From "About IHME":

The Institute for Health Metrics and Evaluation (IHME) is an independent population health research center at UW Medicine, part of the University of Washington, that provides rigorous and comparable measurement of the world's most important health problems and evaluates the strategies used to address them. IHME makes this information freely available so that policymakers have the evidence they need to make informed decisions about how to allocate resources to best improve population health. 

Do you see the word Epidemiology anywhere?  I think that the reason I bristle is not that we shouldn't ask hard questions about models. It is the need to ask whether weak models in a different field should cast doubt on a field of study. I dunno -- do bad IHME models make me ask hard questions about physics too?

What about the director?

Christopher J.L. Murray, MD, DPhil, is the Chair and Professor of Health Metrics Sciences and Director of the Institute for Health Metrics and Evaluation (IHME) at the University of Washington. His career has focused on improving health for everyone worldwide by improving health evidence. A physician and health economist, his work has led to the development of a range of new methods and empirical studies to strengthen health measurement, analyze the performance of public health and medical care systems, and assess the cost-effectiveness of health technologies.

I see Health Economist but not Epidemiologist. There is a department of Epidemiology at the University of Washington and there are people in IHME affiliated with it, but it's hardly a large overlap. It's like blaming the Economics Department for economic forecasting models developed by the Business School. It's the right general disciplinary area and these people might talk, but they are not always going to agree with methodology. But there is definitely not a tight collaboration here and IHME has its own graduate programs in Health Metrics, which it is quite clear are distinct from Epidemiology. They even have their own PhD program in Health Metrics.

Now, I don't want this line of criticism to take away from the fact that the IHME model has some . . . limitations. Here is Ellie Murray (no relation, I think) quoting a statistics link on the IHME model because it was so different than what Epidemiology does. Here is Carl Bergstrom's critique of the model 9UW biology). Here are two bloggers showing discomfort with it a week ago, for many of the reasons that are coming out now.

The IHME model was a very good way of articulating exponential growth. I think Kevin Drum is doing something almost as sophisticated and his closeness to the data has him asking some good questions.  Recent papers have me asking some of the same questions and, coupled with the Iceland data (nicely noted by Korolev), is forcing this (non-epidemic modeling epidemiologist) to ask some hard questions.

That said, the first part of the post was really nice as Ivan Korolev is doing some smart discussion of SEIRD models and is the sort of contribution that matters, as the models that he is critiquing at least come from the field itself and the Iceland data set is very useful adjunct information. Maybe I need to go back and spend some time on that.

Anyway, the health field is complicated and there are a lot of overlapping issues here. I do think it is of great service to be asking hard questions about the models that guide policy. If people want some bad Epidemiology models, I am pleased to provide some examples. But in general some more precision about the field questions would be useful here.