Estimands and targets of inference

This article by Mike the Mad Biologist is worth reading in full (warning: some strong language is included in this piece).  It engages the issues of charter schools in a very detailed way, with a focus on the state MA, which seemed to show fewer issues than many other regions of the country. 

But he also pointed out a very subtle and important point about one of the key findings of the charter school research area:

For long-time readers, I’m going to try (and probably fail) to be polite–just this once, even though I was somehow supposed to craft a coherent response on Twitter after carefully reading technical articles while at work (by the way, never assume someone is unfamiliar with the literature. But I’m getting ahead of myself). The charge, led by Adam Ozimek, focused on a couple of papers by Angrist et al.–though not all of the papers (again, getting ahead of things…). They’re good papers*, and essentially compare students who wanted to attend charters but lost the lottery and wound up in regular public schools (note that ‘unpopular’ charters can’t be assessed with this method).

But that last fragment brings up a really important point.  In the context of popular charter schools, the randomized admissions process is a very good instrument for estimating the average causal effect of the charter school intervention.  There are some issues of composition of classes, that Mark may well comment on, that lead to peer effects.  But this is decent evidence. 

However, it only applies in regions where the charters actually have a waiting list.  This was also really interesting:

However, students in non-urban charter schools do worse than the ‘lottery losers’ in regular public non-urban schools–and with the same effect as the gains in the urban schools. 

Yes, the average causal effect flips for non-urban environments (like the suburbs, for example).  It's obviously not appropriate to take the average causal effect of urban schools and apply it to populations with evidence of a reverse of the causal effect.  It also means we need to do a lot more study to understand when this intervention leads to better (versus worse) outcomes.  It also means that I am now loathe to generalize from the places with waiting lists to those without them (where this random experiment cannot happen). 

So unless there is an issue here that I am missing, this sounds an awful lot like "go slowly and carefully" due to some seriously confusing patterns of average causal effect.  Table 8 is the amazing table, and the effect measure modification is quite startling.  So I think that this adds to the growing body of reasons one can be pro-children and still skeptical about charter schools.