More bad statistics from Freakonomics

[Caveat: It is difficult to distinguish between bad and badly presented statistics. The reader should always allow for the possibility of a sound process terribly explained. It is possible that Dr. Sacerdote made some attempt to control for the factors mentioned below, but, just to put things in context, researchers in numerous fields (including epidemiology) are currently struggling with this problem with limited success. If Sacerdote has actually cracked this nut, Freakonomics is certainly burying the lede.]

Freakonomics Radio is not helping Marketplace's average.

Dubner: Let me let you hear from a different economist, his name is Bruce Sacerdote, he's at Dartmouth. He wanted to know how much this kind of activist-parenting -- if you want to call it that -- actually pays off. And one way to measure this, especially if you're talking about educational achievement -- which is what parents probably care about the most -- is to look at adoption studies, where you can actually measure the impact that a family, that the parents, will have on a kid.

Ryssdal: So what's his thesis, that kids adopted into, I guess, high-education homes will be more likely to go to college, is that the deal?

Dubner: Exactly right. If parents are so important, then parents can take an adopted kid who might otherwise not have gone to college, and that kid will become college material. So Sacerdote sliced and diced a lot of good data and he did find parental influence.

Bruce Sacerdote: But it's not quite as big as I expected to find.

Ryssdal: All right, so quantify for me: how big is not so big?

Dubner: If you're a child who's adopted into a high-education family -- that is where the parents both went to college -- you are about 16 percentage points more likely to go to college than a kid who gets adopted into a low-education family. So that sounds pretty good, OK?

Until you compare that to the rate for biological kids from high-education families, who are about 75 percentage points more likely to go to college than biological kids from low-education families. So on the one hand, this is a little dispiriting for parents. We don't seem to have as much influence as we might think. On the other hand, in a weird way, it kind of takes some of the pressure off, right? At least it did for Bruce Sacerdote.

Sacerdote: This notion that genes are really important and that kids are hard-wired to do certain things, I think understanding that did help me relax and not worry so much that I was going "screw them up" in some terrible way.

One of the concerns I have about sending an economist to do a statistician's job is that economists, by training, have a tendency to make things simpler.* This works reasonably well in economics where the problems are generally too complex to tackle without making some simplifying assumptions about linearity and additive effects and independence and where (perhaps more importantly) the economists have a pretty good idea of which corners can be safely cut. Unfortunately, when they move outside of familiar ground, you often get something like this.

Unless something important has been left out (always a possibility when something is presented to a general audience), Sacerdote has made the common but serious mistake of treating a select group as a random sample. People who choose to adopt are not a random sample of parents, particularly given the difficulties in the process, and those who do self-select have to go a difficult approval process.

Think for a moment about how unrepresentative the resulting sample is. Now consider traits that are likely to be correlated to both parents' and children's education levels and which the process is likely to select for or select out. Here are a few that come immediately to mind:

Teen parenthood;

Unplanned parenthood;

Poverty;

Drug abuse;

Career level (high income may not be a requirement but financial stability is and these days that represents a significant career accomplishment);

Encouragement and support**;

Educational expectations for their children (I can't imagine that all that many prospective parents who don't give a damn about their kid's education make it to the end of the process).

The genetic component of this question is confounded with all these things and more. As a result, (at least as presented here) Sacerdote actually succeeds in proving the opposite of his conclusion. His example doesn't show what, if any, role genetics plays in how likely a kid is to go to college. That 75 could be mainly the result of poverty (which is highly correlated to parent's education levels). I'm not saying that it is but there is no way of ruling out that and other explanations.

What Sacerdote has shown is that even when you control not only for genetics but also for most obvious environmental factors, you still got a 16 point bump just from having adoptive parents who both went to college. Under any circumstances this would be a substantial improvement but in the context of this natural experiment it's huge.

At the risk of over-sharpening the blade, we have here a natural experiment that shows nothing about the role of genetics in academic achievement. What it does show is that even when you greatly reduce most of the obvious parenting-related environmental sources of variability, you still get a substantial effect from the remaining factors. A highly respected researcher then uses this example to argue publicly that genetic factors far outweigh the impact of parenting.

We really need to do something about Freakonomics.

(I'm also not comfortable with the way Sacerdote gives a percentage difference rather than the ranges but that's a subject for another post)


* There are, of course exceptions. Most notably Nate Silver.

** I'm pretty sure in most low education adoptive homes you won't see this relationship:

Hart and Risley also found that, in the first four years after birth, the average child from a professional family receives 560,000 more instances of encouraging feedback than discouraging feedback; a working- class child receives merely 100,000 more encouragements than discouragements; a welfare child receives 125,000 more discouragements than encouragements.