The most open-label of open-label experiments

A big part of being a statistician is knowing when to get nervous, knowing when cutting a small corner could produce massive bleeding. One of those cut corners we pay particularly close attention to is the open-label trial, where the subjects know exactly what treatments they are getting.

In a perfect world, neither the subjects in a trial or the people administering the treatments would ever know who was getting what. This double-blind approach protects us from the placebo effect, which has an unfortunate way of popping up whenever humans are the subjects of research.

At the risk of being obvious, it is next to impossible to perform a double-blind experiment in most areas of educational research -- everyone knows who got the treatment and who didn't -- but that doesn't mean that the underlying reasons for preferring double-blind tests aren't there. We routinely allow for the possibility that the placebo effect can affect pretty much everything from surgery to the immune system. Does ignoring the possibility that it might affect student performance seem like a safe assumption?

If anything, education is a textbook example of an area where we would prefer not to use an open-label approach. We are working with a test population that's highly suggestible and treatments that rely heavily on the subjects' attitude. Under these conditions, telling subjects that they are about to receive a treatment is highly likely to bias the results.

And in the case of charter schools, students aren't just told they are about to receive a treatment; they are often told, in the most dramatic way possible, that they are about to have their lives transformed. Pedagogically, this is a good idea. It helps establish the belief that the student will succeed, a belief that can easily become self-fulfilling. Statistically, though, it greatly muddies the waters.

Watch the first minute of this Sixty Minutes segment. Look at the expressions of the father leaping out of his seat with excitement and the mother and daughter crying with joy. You'd have to be emotionally dead not to empathize with this family's feelings, but you'd also have to be a poor statistician not to wonder if those emotions contributed to student success.



update: I probably should have mentioned that there is reason to suspect that group dynamics may amplify the placebo effect here.

Implications of "counseling out" part II -- missions

This is the second part of a reply to a previous comment about counseling out disruptive students. In the first part I discussed how charter schools' greater freedom to get rid of problem students makes reliable public-to-charter school comparisons difficult, but as important as that question is, there's another that's even more fundamental when discussing these practices: what exactly do we want charter schools to do?

We have two basic choices:

The first is that charter are what all schools should eventually be. In this model, charters provide the template for the American education system. If this is what we're asking from them, then charters' problems dealing with disruptive and non-cooperative students is a serious failure;

If, however, we look at charter schools as niche programs designed to target specific areas and subpopulations, then counseling out student for academic or behavioral reasons may not be a problem at all. If the purpose of these schools is to allow room for experimentation, pump additional resources into under-served areas and provide a better match for certain kids who aren't getting what they should from the one-size-fits-all approach, then counseling out is a necessary part of the model.

Most champions of charter schools would probably pick the first model but most of the major criticisms that have been made recently about charters schools (data biasing issues, accusations of cherry-picking, questions about scalability) largely go away under the second model.

I have a feeling we'll be coming back to this one.

Implications of "counseling out" part I -- metrics

This comment from Michael Bish raises a couple of big questions:

One important point is that there is a big difference between counseling a student out because they have low test scores and counseling them out because they are actively disrupting the education of their peers.

The first issue here involves metrics. The idea that you should build a school system around a handful of test scores is one of the central tenets of the reform movement. Test scores are supposed to drive funding, contracts, allocation of resources, evaluations, bonuses, even terminations, but if you give certain schools more freedom to get rid of disruptive students the whole system breaks down.

Disruptive students not only take up time from the teacher that could be used in instruction; they also make it difficult for students to concentrate and tempt other others to act up. The result is that everyone's test scores go down.

By getting this student transferred you've not only raised the scores of an entire classroom of your students; you've lowered the scores of a comparable number of students in a public school in the same area. Since your school's evaluation, funding, and future contracts are dependent on your performance relative to other schools in the area, you can get a substantial double lift out of that single transfer.

And since public school to a large extent have to work with the students they are given, charter schools always win this one.

Nobody loves an orphan technology

And the loneliest orphan of them all is over-the-air television. It's the kind of story you would expect to find in a Lemony Snicket novel, complete with evil guardian trying to kill it off to get to the family estate (bandwidth).

OTA television has few friends. Cable and broadband get all the attention while the media has been committed to the death of broadcasting/networks for more than three decades now.

Of course, one of the problems with being committed to a narrative is that it forces you to ignore the details that don't support the narrative and these have an unfortunate way of being the interesting ones.

Case in point: this story on the growth of Univision, though a bit credulous ("Univision set to become top U.S. broadcast network"), contains some impressive statistics about the growth of the network. It does not, however, contain any mention of this:

Univision is concerned because nearly 28% of Hispanic households — and 43% of homes where Spanish is the primary language — watch TV only via over-the-air transmissions, according to a 2005 National Association of Broadcasters report to the FCC.

Given that Univision skews toward the Spanish-language only households, that means a big chunk of its audience is coming in over the air. That means that selling off that part of the spectrum would have big consequences for the fast growing area of Spanish language media. That's an awfully important detail to leave out.

"Counseling out"

(This post also appears at Education and Statistics.)

Paul Tough writing in Slate recounts the following:

In Whatever It Takes, in one of the chapters on the Promise Academy middle school, I describe the impact of the KIPP schools in the Bronx and Harlem on the Promise Academy’s leaders and staff. This was during the first few years of the Harlem Children Zone’s middle school, which were a struggle, and those KIPP schools, which had very good test results, were for the Promise Academy administrators both a standard to be aspired to and a frustrating reminder that their own students weren’t performing at the same high level as KIPP’s students.

Terri Grey, the Promise Academy principal at the time, believed the attrition issue was part of what was holding her school back. As she put it to me in one conversation, “At most charter schools, if the school is not a good fit for their child, the school finds a way to counsel parents out”—to firmly suggest, in other words, that their child might be happier elsewhere. “Whereas Promise Academy is taking the most disengaged families and students and saying, ‘No, we want you, and we’re trying to keep you here, and we don’t want to counsel you out.” That policy made it impossible, she believed, for the Promise Academy to achieve KIPP-like results.

I’m not entirely convinced that that was the real problem at Promise Academy—or that the KIPP schools in New York were actually “counseling out” a significant number of students. But I do think it’s true that Geoffrey Canada’s guiding ethic has always been to go out of his way to attract and retain the most troubled parents and students. And that makes running a school, or any program, more difficult, even if it makes the mission purer and, in the end, more important.

For reasons I'll get to later, I suspect that the number of students you have to "counsel out" to have a significant effect on a school's test scores is lower than Mr. Tough realizes, but there are a couple of more important points.

The first is that selective attrition is recognized as a serious issue not just by critics of the reform movement but by responsible people within the charter school community.

The second is that all charter schools and charter school administrators are not interchangeable. There are some gifted educators with great ideas in that system. We've spent almost two decades overlooking the flaws in charter schools. It would be a serious mistake to try to compensate by overlooking the strengths.

Context in posts

This post by Megan McArdle was interesting for the context. I am certain that I will say something I regret sooner or later; it's probably a good thing not to be judged by hasty words.

But I also think it is important to remember that all quotes can be made to look bad out of context. In the "sound-bite" environment of the modern world it is already hard enough to discuss complex issues (*cough* education reform *cough*) without adding in misleading quotes.

It is just something to keep in mind . . .

"they are purging nonperforming students at an alarming rate"

(This post also appears at Education and Statistics.)

Mike at ScienceBlogs has some thoughts about selection by attrition:

A letter to Diane Ravitch from a Los Angeles school prinicipal documents just how dishonest and harmful this practice is (italics [Mike's]):

I received an email from Dr. DeWayne Davis, the principal of Audubon Middle School in Los Angeles, which was sent to several public officials. Dr. Davis said that local charter schools were sending their low-performing students to his school in the middle of the year. He wrote:

"Since school began, we enrolled 159 new students (grades 7 and 8). Of the 159 new students, 147 of them are far below basic (FBB)!!! Of the 147 students who are FBB, 142 are from charter schools. It is ridiculous that they can pick and choose kids and pretend that they are raising scores when, in fact, they are purging nonperforming students at an alarming rate--that is how they are raising their scores, not by improving the performance of students. Such a large number of FBB students will handicap the growth that the Audubon staff initiated this year, and further, will negatively impact the school's overall scores as we continue to receive a recurring tide of low-performing students."

Ravitch concludes:

Doing better than an under-resourced neighborhood school is not the same as getting "amazing results." Very few charters do. Probably less than 5 percent. Charters are not a silver bullet. They are a lead bullet. Their target is American public education.

This is just par for the course for modern conservatism: have private systems skim the cream, and leave the public sector to clean up an impossible mess. When they can't, this supposedly shows the inability of government to solve problems.

I have a few points to add:

1. This is a brutal way to treat these kids. You build their hopes up, then crush them, then dump the kids in a new school in the middle of the year;

2. We are talking about getting an influx of students who are badly behind and who are ready to give up and/or act out. This will disrupt classes slightly less than having a nearby car alarm go off at random times once or twice an hour;

3. But I think Ravitch overstates the case against charter schools. I've dealt with some small, independent schools that have impressed the hell out of me and I can see them playing an important role in our system, though a radically different role than Arne Duncan sees.

Silence

This post from Worthwhile Canadian Initiative is probably the best reason for my recent lack of activity in the blogosphere. Coupled with a lot of travel, I am quite behind.

Hopefully a temporary condition.

In the meantime, Mark has been putting up some great material that is well worth reading, so at least radio silence hasn't struck yet.

Quick note on good stuff

No time to blog this afternoon but if there was I'd join in this discussion on polls and momentum. While you're there, take a minute to check out the comments.

On 'Good Morning America': Worthy defends plan to jail parents who skip school conferences

From The Detroit News:

Wayne County Prosecutor Kym Worthy defended her plan to jail parents who repeatedly miss conferences with their children's teachers during an appearance today on ABC's "Good Morning America."

Worthy said many people think she is looking to simply jail people for three days if they miss a parent-teacher conference, but she said her plan is not that strict. She is calling for the jail stay if a parent repeatedly misses conferences and isn't in touch with teachers and school officials.

Parents of achieving students are exempt and a parent who is in constant contact with the school also is exempt. Worthy said a parent could miss three or four conferences before officials would start looking as to why they were not attending them. Those with medical conditions also would be exempt.

I'm not going to get into the appropriateness or effectiveness of this proposal, but it is another reminder how different the population of charter schools is.

I taught in an inner-city prep school targeting serving lower income students, a school run by some of the most dedicated educators I've ever encountered. They took kids who were performing below grade level (sometimes by more two grades) and got almost all of them into college.

When I say almost all, I mean almost all. The drop-out rate was extremely low. This was possible partly because the administrators made it a priority and partly because schools with admission processes tend to select out many of the most at-risk students. (This is why we expect charter schools to have lower drop-out rates than comparable public schools, and why we are so concerned when we see the opposite.)

But as dedicated and hard working as the faculty and administration of that school was, every one of them would tell you that the engaged and supportive families of the kids made their jobs much, much easier. When there was a parents' night you could expect pretty much one hundred percent attendance and when there was a problem with a student you could pretty much count on a concerned response.

Public school teachers and administrators seldom have those advantages.

And yes, it does make a difference.

More fun and games

I've got a new post up at Education and Statistics on the topological doodling game, Sprouts. If topological doodling strikes you as an enjoyable way to spend your time, I highly recommend that you check this out.

Tax increases as landscape perturbation?

Over at Felix Salmon's site, Justin Fox presents an interesting counter-argument to Greg Mankiw:

[T]here’s a really fascinating tale in [Sam Howe Verhovek’s Jet Age: The Comet, the 707, and the Race to Shrink the World] involving tax incentives. During the Korean War, Congress enacted an excess profits tax meant to keep military contractors from, well, profiteering. In its infinite wisdom, Congress defined excess profits as anything above what a company had been making during the peacetime years 1946-1949.

Boeing was mostly a military contractor in those days (Lockheed and Douglas dominated the passenger-plane business), and had made hardly any money at all from 1946 to 1949. So pretty much any profits it earned during the Korean conflict were by definition excess, and its effective tax rate in 1951 was going to be 82%. This was unfair and anti-business. If similar legislation were enacted today, you could expect U.S. Chamber of Commerce members to march on Washington and overturn cars on the streets.

It being 1951, Boeing instead sucked it up and let the tax incentives inadvertently devised by Congress steer it toward a bold and fateful decision. CEO Bill Allen decided, and was able to persuade Boeing’s board, to plow all those profits and more into developing what became the 707, a company-defining and world-changing innovation. Writes Verhovek:

Yes, it was a huge gamble, but for every dollar of the dice roll, only eighteen cents of it would have been Boeing’s to keep anyway. For Douglas and Lockheed, both in a much lower tax bracket, that was not so easy a call.

So that’s it! High tax rates—confiscatory tax rates—spur innovation! Well, at least once in a blue moon they do. Which is an indication that there might be some important stuff missing from the classic economists’ view of taxation, as summed up by Greg Mankiw a few weeks ago:

Economists understand that, absent externalities, the undistorted situation reflects an optimal allocation of resources. It is crucial to know how far we are from that optimum. To be somewhat nerdy about it, the deadweight loss of a tax rises with the square of the tax rate.

Somehow I don’t think that formula held true in Boeing’s case.

This leads me to wonder if this reminds anyone else of algorithms that locate superior optima by slightly perturbing fitness landscapes (processes closely related to simulated annealing). Mankiw complains that certain taxes distort the economic landscape, but if local optimization is an issue (as was apparently the case with Boeing), then mild distortion from time to time is likely to lead to a better performing economy.

For background, here's an excerpt from a post on landscapes. The subject was lab animals but the general principles remain the same:

And there you have the two great curses of the gradient searcher, numerous small local optima and long, circuitous paths. This particular combination -- multiple maxima and a single minimum associated with indirect search paths -- is typical of fluvial geomorphology and isn't something you'd generally expect to see in other areas, but the general problems of local optima and slow convergence show up all the time.

There are, fortunately, a few things we can do that might make the situation better (not what you'd call realistic things but we aren't exactly going for verisimilitude here). We could tilt the landscape a little or slightly bend or stretch or twist it, maybe add some ridges to some patches to give it that stylish corduroy look. (in other words, we could perturb the landscape.)

Hopefully, these changes shouldn't have much effect on the size and position of the of the major optima,* but they could have a big effect on the search behavior, changing the likelihood of ending up on a particular optima and the average time to optimize. That's the reason we perturb landscapes; we're hoping for something that will give us a better optima in a reasonable time. Of course, we have no way of knowing if our bending and twisting will make things better (it could just as easily make them worse), but if we do get good results from our search of the new landscape, we should get similar results from the corresponding point on the old landscape.


* I showed this post to an engineer who strongly suggested I add two caveats here. First, we are working under the assumption that the major optima are large relative to the changes produced by the perturbation. Second our interest in each optima is based on its size, not whether it is global. Going back to our original example, let's say that the largest peak on our original landscape was 1,005 feet tall and the second largest was 1,000 feet even but after perturbation their heights were reversed. If we were interested in finding the global max, this would be be a big deal, but to us the difference between the two landscapes is trivial.

Nothing like a good graph to convey bad news

There's an extensive and erudite discussion of this over at Economist's View, with voices ranging from Krugman to Mankiw.

Nate Silver tries to disabuse political reporters of another favorite myth

In today's 538, Nate Silver takes down the myth that polls tend to show momentum. He left out some details that OE readers might be curious about (but, of course, we're not exactly the target audience), but it's an excellent piece of statistical writing.

The Misunderstanding of Momentum

By NATE SILVER

Turn on the news or read through much of the analysis put out by some of our friends, and you’re likely to hear a lot of talk about “momentum”: the term is used about 60 times per day by major media outlets in conjunction with articles about polling.

When people say a particular candidate has momentum, what they are implying is that present trends are likely to perpetuate themselves into the future. Say, for instance, that a candidate trailed by 10 points in a poll three weeks ago — and now a new poll comes out showing the candidate down by just 5 points. It will frequently be said that this candidate “has the momentum”, “is gaining ground,” “is closing his deficit,” or something similar.

Each of these phrases are in the present tense. They create the impression that — if the candidate has gone from being 10 points down to 5 points down, then by next week, he’ll have closed his deficit further: perhaps he’ll even be ahead!

There’s just one problem with this. It has no particular tendency toward being true.

Read the rest...

A probabilistic mystery from Alfred Hitchcock

Don't believe me? See for yourself.