The most open-label of open-label experiments

A big part of being a statistician is knowing when to get nervous, knowing when cutting a small corner could produce massive bleeding. One of those cut corners we pay particularly close attention to is the open-label trial, where the subjects know exactly what treatments they are getting.

In a perfect world, neither the subjects in a trial or the people administering the treatments would ever know who was getting what. This double-blind approach protects us from the placebo effect, which has an unfortunate way of popping up whenever humans are the subjects of research.

At the risk of being obvious, it is next to impossible to perform a double-blind experiment in most areas of educational research -- everyone knows who got the treatment and who didn't -- but that doesn't mean that the underlying reasons for preferring double-blind tests aren't there. We routinely allow for the possibility that the placebo effect can affect pretty much everything from surgery to the immune system. Does ignoring the possibility that it might affect student performance seem like a safe assumption?

If anything, education is a textbook example of an area where we would prefer not to use an open-label approach. We are working with a test population that's highly suggestible and treatments that rely heavily on the subjects' attitude. Under these conditions, telling subjects that they are about to receive a treatment is highly likely to bias the results.

And in the case of charter schools, students aren't just told they are about to receive a treatment; they are often told, in the most dramatic way possible, that they are about to have their lives transformed. Pedagogically, this is a good idea. It helps establish the belief that the student will succeed, a belief that can easily become self-fulfilling. Statistically, though, it greatly muddies the waters.

Watch the first minute of this Sixty Minutes segment. Look at the expressions of the father leaping out of his seat with excitement and the mother and daughter crying with joy. You'd have to be emotionally dead not to empathize with this family's feelings, but you'd also have to be a poor statistician not to wonder if those emotions contributed to student success.



update: I probably should have mentioned that there is reason to suspect that group dynamics may amplify the placebo effect here.