More on the whole "Epidemiology is Hard" thing

There's a new study that's been getting a lot of press:

For generations of Americans, it was a given that children would live longer than their parents. But there is now mounting evidence that this enduring trend has reversed itself for the country’s least-educated whites, an increasingly troubled group whose life expectancy has fallen by four years since 1990.
Researchers have long documented that the most educated Americans were making the biggest gains in life expectancy, but now they say mortality data show that life spans for some of the least educated Americans are actually contracting. Four studies in recent years identified modest declines, but a new one that looks separately at Americans lacking a high school diploma found disturbingly sharp drops in life expectancy for whites in this group. Experts not involved in the new research said its findings were persuasive.

The reasons for the decline remain unclear, but researchers offered possible explanations, including a spike in prescription drug overdoses among young whites, higher rates of smoking among less educated white women, rising obesity, and a steady increase in the number of the least educated Americans who lack health insurance.

The steepest declines were for white women without a high school diploma, who lost five years of life between 1990 and 2008, said S. Jay Olshansky, a public health professor at the University of Illinois at Chicago and the lead investigator on the study, published last month in Health Affairs. By 2008, life expectancy for black women without a high school diploma had surpassed that of white women of the same education level, the study found.

White men lacking a high school diploma lost three years of life. Life expectancy for both blacks and Hispanics of the same education level rose, the data showed. But blacks over all do not live as long as whites, while Hispanics live longer than both whites and blacks. The decline among the least educated non-Hispanic whites, who make up a shrinking share of the population, widened an already troubling gap. The latest estimate shows life expectancy for white women without a high school diploma was 73.5 years, compared with 83.9 years for white women with a college degree or more. For white men, the gap was even bigger: 67.5 years for the least educated white men compared with 80.4 for those with a college degree or better.
...
The decline among the least educated non-Hispanic whites, who make up a shrinking share of the population, widened an already troubling gap. The latest estimate shows life expectancy for white women without a high school diploma was 73.5 years, compared with 83.9 years for white women with a college degree or more. For white men, the gap was even bigger: 67.5 years for the least educated white men compared with 80.4 for those with a college degree or better.

This is, unquestionably, a troubling finding and I have every reason to believe that the researchers did a responsible job. None the less, this part still troubles me:

Researchers said they were baffled by the magnitude of the drop. Some cautioned that the results could be overstated because Americans without a high school diploma — about 12 percent of the population, down from about 22 percent in 1990, according to the Census Bureau — were a shrinking group that was now more likely to be disadvantaged in ways besides education, compared with past generations.

Professor Olshansky agreed that the group was now smaller, but said the magnitude of the drop in life expectancy was still a measure of deterioration. “The good news is that there are fewer people in this group,” he said. “The bad news is that those who are in it are dying more quickly.”

Dying at the age of seventy in 1990 would mean you were born in 1920. Dying at the age of sixty-seven in 2008 would mean you were born in 1941. If you were to build a model to predict whether someone born in in 1920 would finish high school, it would certainly look different than a model predicting the same thing for someone born in 1941. We know this, for one reason, because the target variable was much less frequent for the second group.

Of course, we're talking about averages and distributions and that can complicate things in a number of ways (most of which don't make it into a newspaper account of a research paper), but no matter how you slice it, what it meant to have a high school diploma changed greatly from the early to the middle part of the Twentieth Century, and that change is very difficult to control for.

I think it's generally a good, conservative rule to assume that when the relative size of a segment of the population drops dramatically, the composition of the segment is likely to shift. Thus a sharp increase in a behavior in these groups' behavior may simply be a result of the people who didn't show the behavior not being in the group any longer.

I'm not saying that this is the case here. I'm just saying that this sort of analysis makes me  nervous.

(I have the same problem with dropping poll responses, but more on that later