A different take on grades and money from home -- part 1 Hypothesis Shopping

New study claims more money from home is associated with lower grades.

I came across this in a post from Andrew Gelman who had some sharp criticisms for the analysis that lead to this conclusion (criticisms seconded by Joseph in these previous posts). I clicked through the links with a slightly bloodthirsty attitude, already toying with ideas for posts mocking Dr. Hamilton, the hapless researcher. As I followed through, though, I felt less and less like mocking and more inclined to a shaded, even positive view, particularly given my previously noted feelings about looking at research in context.

Not that I disagree with Andrew and Joseph's criticisms -- the survivor bias is really difficult to get past -- but on some big how-we-do-science questions, I see a lot to like here, starting with where Hamilton started (and where she didn't).

Back in 2004, Hamilton and a group of other researchers spent a year studying a group of mostly first year students at large university, then continued to track their progress and interview them for the next five years. One of the things they observed was that the students who didn't have to work or struggle financially due to checks from home tended to take school less seriously, study less and get worse grades. Hamilton decided to follow this up by seeing if this pattern held nationally.

The resulting analysis was not well done, but I like the general approach: use case studies, participant observation, interviews and similar techniques to study your subjects extensively, form your hypotheses based on those observations, see how they hold up when tested against more general data. (or, put more broadly, actually devote time and effort into coming up with your hypotheses.)

That may seem like damning with faint praise, but I think a lot about where hypotheses come from and I give quite a bit of credit to researchers who put the work in to do it right, particularly in an age of hypothesis shopping.

Hypothesis shopping is one of our two leading sources of bad studies. The other is the epicyclism. The first entails running through endless unlikely relationships until one turns up significant (sometimes mistaken for data mining by people who know nothing about data mining). The second entails coming up with increasingly convoluted hypotheses to fit the data, often to preserve an ideological position. Sadly there's a nontrivial overlap between the two.

Hypothesis shopping has always been around but only recently has it become what you might call "cost effective" due to huge advances in the availability of data and the power of computers. Today, anyone with a large data set and a late-model laptop can crank through thousands of possible relationships looking for something with a good p-value. Just start with a couple dozen potential dependent variables, a few hundred independent variables, and some reasonable sounding transformations and interactions. You are pretty much certain to find something at least as impressive and "significant" as the the findings you'll routinely see in Slate or the New York Times.

The results of these processes are often absurd enough to be obvious to everyone (with the exception of the aforementioned publications), but for every watching-sports_before-conception-makes-you-more-likely-to-have-a-boy (I really hope that's a made up example), there's a provocative but not easily dismissed story with huge policy implications. It would be useful to know if we're talking about an effect that has been observed in other contexts or if it's just the green jelly bean.

Knowing the provenance of a hypothesis doesn't protect us from bad research but it does, to a large degree, inoculate us against certain kinds of bad research.