I haven't had time to go through the Boston report in detail, but assuming that it's solid (and I have no reason to think otherwise), there are still a couple of reasons to be careful about drawing overly broad inferences from this particular city at this particular point in history.
The first (which is pretty much self-evident) has to do with Boston's unique relationship with higher education. The second, and more important, goes to the heart of the problems with modern urban schools.
Why wouldn’t parents try to get their children into charters that would get the kids on track to go to college? One explanation is that the debate about whether charter schools “work,” with its focus on testing and college placement, loses sight of the many reasons why people choose a school and what they value in an education. Charter schools can be set up for any purpose. Some focus on the arts, others emphasize cultural heritage (there are multiple Hmong charter schools in the Twin Cities alone); some are vocational, others rigorously academic. As Rockoff puts it, asking whether a particular school is good based on test scores or college placement is in many cases the wrong question: “To extend the restaurant metaphor, some people like Italian, others like Thai food. Similarly, many [charter] schools focus on tested subjects, while others might emphasize creative writing or the arts.” Not every charter school is right for every kid. Nor would it be an unmitigated good if all parents could segregate their children by their interests and beliefs.But a big reason, maybe the big reason, that urban parents send their children to charter/magnet and parochial schools is left out of that paragraph. In general, parents want their children to be successful happy and safe, but of those, safety is the most immediate. These non-standard schools usually have a low or no-tolerance approach to gangs. More significantly, there's a huge anti-gang selection effect.
If you want to compare different urban education initiatives, you have to take into account gang activity and neighborhood violence, not just in terms of absolute levels but in terms of attitudes and expectations, and, while I have no expertise in the area, I do know enough to know that understanding urban crime in Boston over the past twenty years, a period that saw both the advent and apparent collapse of the "Boston Miracle," is something that does require considerable expertise.
None of this undercuts the findings of the study, but it does greatly complicate the process of drawing inferences from it. It's possible that what worked in Boston won't work as well in most urban areas. It's also possible that what worked there will work even better elsewhere. I'm not being cute here -- I could honestly see this going either way -- I'm just trying to emphasis just how messy this data is.
We have here everything your stat professor warned you about. Interactivity. Non-linearity. Heteroskedasticity. Nesting. Selection effects. Missing not at random. Falsification. Hawthorne effects. Volunteer effects. Peer effects. Researcher bias. Sample size issues.
Though I'm not a Bayesian by training, I think the appropriate response is at least in spirit, Bayesian. We will have to factor our doubts about the research in when we weigh our conclusions but that's not the same as ignoring those conclusions entirely.
This brings me back to what I liked about Fisman's article. It's balanced, it acknowledges the limitations of the data, and it suggests focus on areas where reforms will do the most good if their assumptions are sound and will do the least damage if they aren't.
But more on that later.