Charter Schools and Hedge Funds

A Wall Street trader draws an interesting analogy:

Survivorship bias
In statistics, this is a textbook case of what is known technically as “survivorship bias”—otherwise known as “ignoring data that makes you look bad.” This statistical fallacy opens up enormous opportunities for people with flexible ethics and an entrepreneurial bent. It is a mainstay of the hedge fund industry, for instance.  Hedge fund indices routinely appear to outperform simple, non-fee generating investment strategies like index funds by neglecting to include funds that closed down, and take funds out of the index whenever they stop reporting performance (hint: no fund fails to report good performance!).

R u hot or not?
Some funds, which shall remain nameless on the advice of counsel, improve quite a bit on these crude methods. They recruit new traders continuously, and give them a bit of capital. After a few months, the ones that make money are trumpeted to investors as the next hot thing, while the ones that don’t are quietly fired and never mentioned again. In this business model, what the traders actually do is irrelevant, as long as they are cheap and willing to sign leonine contracts. In fact, buying monkeys and letting them flip coins to predict market direction would work even better, though this might make the game a bit too obvious.

Repeat as necessary
The parallels to the emerging charter school paradigm are obvious. You start a lot of charters. Some do better than public schools, some do worse, but overall they underperform. You shut down the bad ones. Now repeat the analysis with the non-terrible ones. Improvement!  Except that the students unfortunate enough to attend these terrible school don’t just disappear (we hope – let’s not give charter schools any ideas!). However, they do disappear from the CREDO study, and that seems to be good enough for the charter sector and its advocates to proclaim this a success story.

I haven't seriously dug into the CREDO data so I can't vouch for the rest of the trader's analysis but it looks reasonable and it reinforces a point I've been making for a while: there are a lot systemic factors which favor charter schools and which furthermore tend to bias the data in these schools' favor. In other words, given the conditions charters operate under, we would expect them to outperform public schools even if they were pedagogically identical due to factors like selection, volunteer and placebo effects and even if the schools were performing identically, we would expect the charters to look better due to factors like attrition and survivorship. Add to this some non-trivial cases of data cooking and you can see where this leads.

Even with these factors, charter school performance has been tepid, at best. I don't see that as an argument for abandoning the experiment, but it does mean that the standard the-more-charters-the-better narrative is no longer viable and movement reformers like Arne Duncan have got to mark to market if they expect to be taken seriously.