Thoughts on Ray Fisman's much alluded-to charter school article

As mentioned before, Ray Fisman has a piece up at Slate discussing charter schools and it's quite good. Arguments for both sides are spelled out in a clear and balanced manner and, with a few exceptions (and only one of those major), data issues are addressed as thoroughly as is possible in a general interest publication.

That major issue is cheating and data manipulation and it's the one problem I would most like to get Fisman's take on, partly because I really do believe it's probably the biggest challenge facing the reform movement but mainly because of Fisman's expertise in the question of how best to set up metrics and incentives in order to keep things running smoothly.

My other issues are more the sort of technical points you probably couldn't get away with in Slate. For example, take a look at the following from Fisman:

At least part of the disagreement revolves around whether charter schools deliver on their promise to improve student outcomes. You might think this is a relatively easy proposition to evaluate—just compare whether charter school kids do better on tests than those in public schools. But any effort to compare performance is confounded by the fact that the kinds of parents who take steps to enroll their children in charter schools may be the kind of motivated and supportive parents whose children would have done just fine in any school system. (In the current study, charter school applicants do in fact have higher than average test scores even before they enroll. However, other analyses have seen charter school applicants with below-average scores, perhaps because kids struggling in the public system are more apt to look for other options.) And if the longer hours and additional school days that are a feature of many charter schools lead underperformers to drop out, the select group that remains may again be made up of those who would have tested well in any school environment.

This isn't just a question of underperformers. Based on my classroom and tutoring experience, I strongly expect we're seeing a strong interaction between treatment and selection. Different kids respond differently to different approaches. There's a very good chance that those kids (and families) who volunteer for more time and more work are also the kids who respond best to that approach. This is not an argument against charter schools; it is, however, an argument for caution in generalizing results.

A somewhat more serious issue shows up in the next paragraph

But for at least a subset of charter schools, researchers can come fairly close to running a clinical trial where some applicants are enrolled at charters and others are left in the public system purely by chance. The reason is that many charter schools are oversubscribed, and their scarce spots are allocated through a lottery. So whether a particular student gets assigned a slot at the charter school is luck of the draw. (This randomness in gaining admission to sought-after charter schools has been documented recently in the films Waiting for Superman and The Lottery.)

Numerous studies have used this lottery method to analyze the impact of charter schools on standardized test scores, and by and large they report similar findings: Charters in rural or suburban areas don’t do any better than public schools, while in urban areas they are associated with greater test score improvements in math and language. But another important point from past studies is that there is enormous variation in the effectiveness of charter schools. There are some great ones but also some real duds.

Normally, in clinical trials, we can treat our subjects as independent. After all, my odds of developing a side effect should have nothing to do with whether or not you developed that side effect. There is simply no way you can assume that the behavior of one student is independent of the rest of the class and that lack of independence can be extraordinarily difficult to account for. While useful, the lottery method is a long way from giving us the results we would expect from a good clinical trial.

But, that aside, Fisman's point about urban schools is tremendously important and is explored in more detail later:

Some charter school advocates will surely point to the new study as yet more evidence that public school districts should be replaced by a more decentralized approach to education, with a greater emphasis on charter schools. But that’s a very narrow interpretation of the mounting evidence of charter schools’ successes. Focusing on these successes glosses over the many cases where charter schools fail to outperform their public peers. For suburban districts in Massachusetts, for example, the numbers don’t favor the charter school advocates, in large part because suburban public schools in the state are pretty effective already.

This is a huge point and it addresses one of the biggest concerns of reform skeptics, namely that real problems might be used as cover to change the parts of the system that are working but which are ideologically disagreeable to some of the reformers and, as a result, leave a system more broken than it is now.

From the beginning of the debate, the question of how to close the achievement gap has gotten conflated with the question of general education reform. The wretched state of education in high poverty areas was often used to argue that we needed to make immediate sweeping changes to all areas of education and some of those changes were truly radical.

Until recently, many of the reform proposals (firing 80% of new teachers, extensive privatization, effectively removing teachers' right to unionize, and quite a few more) allowed for little if any middle ground, particularly given the amount of data that was not breaking in the reformers' favor. I strongly suspect that Diane Ravitch's Road-to-Damascus conversion came from her inability to move gradually away from her previous positions.

I realize I'm reading a lot into one recent article, but this seems to be where serious reformers like Fisman and Jonah Rockoff are now:

1. Charter schools have a great deal of promise;

but

2. They aren't right for all kids or all situations;

3. and there may be scaling limitations both in terms of teachers and parents;

4. so we should focus on areas where the need is greatest and the evidence for effectiveness is strongest.

This is not the exact list I would have made (OK, technically I did make this list but you know what I mean), still there's nothing here that I would disagree with. What we have now are questions of degree, which tend to be more productive and are much more likely to be convergent.

Which would be nice for a change.