The infection fatality rate is not a single number

This is Joseph.

So I was reading 2 preprints on covid-19 infection fatality rates (IFR). The first one concludes:

Based on a systematic review and meta-analysis of published evidence on COVID-19 until the end of April, 2020, the IFR of the disease across populations is 0.75% (0.49-1.01%). However, due to very high heterogeneity in the meta-analysis, it is difficult to know if this represents the ‘true’ point estimate. It is likely that different places will experience different IFRs. More research looking at age-stratified IFR is urgently needed to inform policy-making on this front.

The second one concludes:

The infection fatality rate of COVID-19 can vary substantially across different locations and this may reflect differences in population age structure and case-mix of infected and deceased patients as well as multiple other factors. Estimates of infection fatality rates inferred from seroprevalence studies tend to be much lower than original speculations made in the early days of the pandemic. 

So let me start with one point that I think is true: IFR varies from place to place, for reasons that we don't fully understand. I am not sure that public transit is a great candidate, but maybe I am wrong about that. I personally think that long term care facilities are a big piece of it, but all of this thinking is non-causal in nature.

But, given that caveat, a general sense of the lethality of the disease is useful. Pre-print #1:

The meta-analysis demonstrated a point-estimate of IFR of 0.75% (0.49-1.01%) with high heterogeneity 

 Preprint #2:

Infection fatality rates ranged from 0.03% to 0.50% and corrected values ranged from 0.02% to 0.40%.

 So these are quite different estimates. One big difference is the that the second report did not include any of the government reports. But in hard hit places, the IFR is reaching 0.19% (New York) and 0.16% (Lombardy).

The clue to the difference is when the second pre-print talks about New York:

Massive deaths of elderly individuals in nursing homes, nosocomial infections, and overwhelmed hospitals may also explain the very high fatality seen in specific locations in Northern Italy and in New York and New Jersey. A very unfortunate decision of the governors in New York and New Jersey was to have COVID-19 patients sent to nursing homes. Moreover, some hospitals in New York City hotspots reached maximum capacity and perhaps could not offer optimal care. With large proportions of medical and paramedical personnel infected, it is possible that nosocomial infections increased the death toll. Use of unnecessarily aggressive management (e.g. mechanical ventilation) may also have contributed to worse outcomes. Furthermore, New York City has an extremely busy, congested public transport system that may have exposed large segments of the population to high infectious load in close contact transmission and, thus, perhaps more severe disease. A more aggressive viral clade has also been speculated, but this needs further verification. These factors may explain why preliminary press-released information on a seroprevalence survey in New York State suggests a much higher IFR. With 20% estimated crude seroprevalence in New York City, including a range between 17.3% in Manhattan to 27.6% in Bronx (adjusted seroprevalence figures have not been released), IFR would be as high as 0.8% in Bronx and 1% in Queens, and even higher if probable COVID-19 deaths are included in the calculation.

 Ok, so now we see a estimate much higher than that seen in the results section. Why? The author conjectures possibilities: poor decisions, hospital transmission, public transit, hospitals were bad, and maybe the virus was different. I kind of think that "we don't know" would be the best summary, but no harm in considering possibilities. But it is where we go next that reveals the real difference:

Moreover, even in these locations, the IFR for non-elderly individuals without predisposing conditions may remain very low. E.g. in New York City only 0.6% of all deaths happened in people

This quote connects to another oddity of the second pre-print: they use studies of blood donors (i.e. very healthy and typically younger individuals). The author argues that estimates from NY are contaminated by people over 65 and with pre-existing conditions (or at least that is how I interpret his comments). This is critical as a study that concludes the IFR ranges from "0.02% to 0.40%" probably has to deal with the population fatality rate of 0.19% and rising.

And so we get back to the title of the post. IFR is an average across many strata, of age, sex, health condition, and health care. The overall summary statistic is not especially meaningful to any specific person stripped of this context. Notice both studies point out heterogeneity in their estimates and that makes sense -- there are different care, demographic, and health characteristics in different populations (as well as the stochastic feature of who happens to get infected first). So there is a big limitation to global estimates of covid-19 IFR,

But the real limitation of preprint #2 is that they are asking a different question than what the title and abstract of the paper appear to imply: "what is the IFR among young, healthy, adults with good access to care". That explains the three blood donor studies (Denmark, Scotland, and the Netherlands) and this has been a central critique. What is really missing is a clear statement of the research question as IFR among the young and healthy.

The first paper states their objective (poorly placed in the paper) as:

This paper presents a systematic effort to collate and aggregate these disparate estimates of IFR using an easily replicable method. While any meta-analysis is only as reliable as the quality of included studies, this will at least put a realistic estimate to the IFR given current published evidence.

 The second paper had a clear objective:

To estimate the infection fatality rate of coronavirus disease 2019 (COVID-19) from data of seroprevalence studies.

 But it was missing a very important caveat: "among young and healthy individuals". In that context, the results from both studies make sense. If anything, this is a lesson that the population structure is the really interesting part. Obviously, a 50 year old with hypertension might have a couple of decades of life left. The CDC estimates that 37% of adult men between 40 and 59 have hypertension. So the estimate among the young and healthy is useful, but hardly can drive policy for the population as a whole.

One also thinks these caveats are needed here too, because this appears to be accurate only if the "young and healthy" caveat is included:

However, it is helpful to know that SARS-CoV-2 has relatively low IFR overall and that possibly its IFR can be made even lower with appropriate, precise non-pharmacological choices.

 Whereas I think pre-print #1 is uncontroversial in their conclusions:

Based on a systematic review and meta-analysis of published evidence on COVID-19 until the end of April, 2020, the IFR of the disease across populations is 0.75% (0.49-1.01%). However, due to very high heterogeneity in the meta-analysis, it is difficult to know if this represents the ‘true’ point estimate. It is likely that different places will experience different IFRs. More research looking at age-stratified IFR is urgently needed to inform policy-making on this front.

 So, I think that what we are learning is that we need to be extremely precise in what we are measuring and including in our analysis of something with a complex structure like IFR. New York City might end up being a unique case, or it could be a harbinger of things to come elsewhere (especially if the speculation about strains of virus are true).

I suspect that sort of more complicated analysis is needed to really understand the policy risks of different pandemic mitigation strategies. I think what is most needed are nuanced estimates that transparently engage uncertainty and population differences. Hopefully both of these preprints will mature in the peer review process.