Diane Ravitch vs. "Superman"

There are few places where the divide in the educational debate between popular perception reality is as depressing as the reaction to Waiting for Superman. Here Diane Ravitch (whose resume includes the first Bush administration and the Hoover Institute) lists some of the distortions, omissions and factual errors.

More on this later.

"The Freakonomics Fallacy"

From an excellent post by Barbara Kiviat (pay close attention to the last sentence):

There’s been some interesting discussion in response to my earlier post about why we expect too much from economists, although a lot of the comments miss my larger point. What I was trying to say is that economics might not entirely be up to the task of explaining what we generally consider to be economic phenomena because we are overconfident about what the discipline has the ability to account for. You might call this the Freakonomics Fallacy. Whatever it is in the world we are trying to explain—crime, climate change, test scores—economics has the answer.

If this isn’t in fact true, then why would we think it? Again, to underscore a part of my earlier argument that I probably didn’t make forcefully enough: we think economics has all the answers because economics has become our major mode of understanding the social world around us. Charities are social businesses. Policy makers are cost-benefit analyzers. Education is a market.

This has not always been the case. In earlier eras, we were often more likely to understand human behavior and social dynamics through other prisms, such as political science, sociology, psychology, anthropology or biology. Indeed, for better or for worse, many of these fields took a turn being the dominant social science. I’m not saying that one frame gives a more accurate or useful picture of the world than another. Just that each leads to a different way of understanding why things happen the way they do because each comes with its own set of assumptions and simplifications.

You can see my thoughts on the subject here.

Children and Placebos

From:

Greater Response to Placebo in Children Than in Adults: A Systematic Review and Meta-Analysis in Drug-Resistant Partial Epilepsy

(published in PLoS Medicine)

The researchers searched the literature for reports of RCTs on the effects of antiepileptic drugs in the add-on treatment of drug-resistant partial epilepsy in children and in adults—that is, trials that compared the effects of giving an additional antiepileptic drug with those of giving a placebo by asking what fraction of patients given each treatment had a 50% reduction in seizure frequency during the treatment period compared to a baseline period (the “50% responder rate”). This “systematic review” yielded 32 RCTs, including five pediatric RCTs. The researchers then compared the treatment effect (the ratio of the 50% responder rate in the treatment arm to the placebo arm) in the two age groups using a statistical approach called “meta-analysis” to pool the results of these studies. The treatment effect, they report, was significantly lower in children than in adults. Further analysis indicated that this difference was because more children than adults responded to the placebo. Nearly 1 in 5 children had a 50% reduction in seizure rate when given a placebo compared to only 1 in 10 adults. About a third of both children and adults had a 50% reduction in seizure rate when given antiepileptic drugs.

...

These findings, although limited by the small number of pediatric trials done so far, suggest that children with drug-resistant partial epilepsy respond more strongly in RCTs to placebo than adults. Although additional studies need to be done to find an explanation for this observation and to discover whether anything similar occurs in other conditions, this difference between children and adults should be taken into account in the design of future pediatric trials on the effects of antiepileptic drugs, and possibly drugs for other conditions. Specifically, to reduce the risk of false-negative results, this finding suggests that it might be necessary to increase the size of future pediatric trials to ensure that the trials have enough power to discover effects of the drugs tested, if they exist.

Of course, with open-label experiments, this should lead to false positives.

And a final word on the subject from America's most successful philosophy major

But I quit that! I've quit ALL drugs. Well... let me say one thing: I twisted my ankle this morning, and I was in quite a bit of pain... so I went to the doctor, and I asked him to give me some pain pills. And he didn't want to do it, but I talked him into it. So he gave me some pills -- and I shouldn't have done this, but I took some about an hour before the show tonight, and right now... I am high... as a KITE! I mean, it is unbelievable! And I would NEVER say this to you people, but, in this case: if you EVER get a chance, to take these drugs... DO IT! They're called... [ he glances from side-to-side cautiously ] Placebos! I mean, I'm thinking that right now I have NO idea where I am at all! It is WILD! [in a hushed voice] PLUH-SEE-BO!

Steve Martin on SNL

Placebos

This is relevant to Mark's posts on how comparisons between students who win and lose charter school lotteries are similar to open label drug trials. Andrew Gelman quotes Kaiser Fung stating that (in drug trials) the standard comparison is:

Effect on treatment group = Effect of the drug + effect of belief in being treated
Effect on placebo group = Effect of belief in being treated

I think that the mathematical formulation of Mark's concern about using charter school lotteries as follows:

Effect in charter school group = Effect of the charter school intervention + effect of belief in being treated + baseline

Effect on students in standard schools = baseline - (any discouragement effect from losing the lottery)

So if the effect in the charter school students is greater than that in students who lose the lottery and are placed in standard schools, that skill doesn't separate the two sources of variation (placebo versus innate effect of the intervention).

The real test would be to introduce some sort of placebo intervention (a charter school that used standard educational approaches??). But that is not an easy thing to accomplish, for obvious reasons. I suspect that this is why psychologists end up having to use deception as part of their studies, despite the ethical quagmires this produces.

UPDATE: missing link fixed

Worth a quick look

Very worthwhile read on the nuances of economics. In particular:

"You see, in a capitalist economy, wealth and well-being are supposed to redistribute to everyone. As capital is allocated and risk managed more efficiently, more opportunities are created for all. Wealth and well-being are supposed to become less concentrated in the hands of a few and more dispersed to the hands of the (often more and increasingly) productive many. Much of Wealth of Nations is devoted to describing the instances and the conditions under which this seemed to be occurring as the economy of Great Britain transitioned from feudalism to one of commercial exchange, industrial production, and small business owners."

H/t: Mark Thoma

Diffferences in Professional Standards

I find it fascinating how the rules change between disciplines. In Epidemiology, the idea of a supervisor being an author on papers resulting from a doctoral discipline is completely non-controversial (and standard practice).

I think it is the difference in resources. The question is asked:

If, as an undergraduate, you handed in a paper that had partially been written by someone else, you would be sent down to the Dean's office, read the riot act with regard to academic integrity, and (probably) given an automatic F in the course.

Why doesn't the same thing happen with respect to a PhD thesis?

In Epidemiology the answer is that there would be no PhD theses under this rubric. Data takes too long to collect and analyze. Furthermore, granting agencies are simply unwilling to give massive amounts of grant funding to a PhD student. All of our research is deeply collaborative and single authored work can only happen in areas quite abstracted from reality or for very senior people who have access to data in a way that others do not.

A war with Iran would look nothing like WW II

When someone says something outrageous and stupid, there's a tendency (particularly on the blogosphere) to lock in on the outrageous part and give the stupidity a pass. Today's case in point comes from David Broder (via a suitably appalled Mark Thoma):

What else might affect the economy? The answer is obvious, but its implications are frightening. War and peace influence the economy.

Look back at FDR and the Great Depression. What finally resolved that economic crisis? World War II.

Here is where Obama is likely to prevail. With strong Republican support in Congress for challenging Iran's ambition to become a nuclear power, he can spend much of 2011 and 2012 orchestrating a showdown with the mullahs. This will help him politically because the opposition party will be urging him on. And as tensions rise and we accelerate preparations for war, the economy will improve.

I am not suggesting, of course, that the president incite a war to get reelected. But the nation will rally around Obama because Iran is the greatest threat to the world in the young century. If he can confront this threat and contain Iran's nuclear ambitions, he will have made the world safer and may be regarded as one of the most successful presidents in history.

Broder is, of course, suggesting just that, but while everyone is reacting to the indecency or, like Dean Baker, pointing out that infrastructure spending has the same effect, no one seems to be focusing on the stupidity of the analogy. So here I go in handy list form:

1. We don't have a draft. This war would mainly be fought with the forces at hand with minimal impact on unemployment;

2. WW II required a massive build-up in our air force and navy. How many more carriers are we going to need if we go war with Iran? In other words, this war would do little to absorb excess capacity;

3. We had a tightly controlled economy during the war that created a build-up in consumer demand. How likely is it that a GOP Congress would go along with that?

4. Immediately after the war, we extended a huge and unprecedented social safety net for the returning forces. Any chance of that happening?

If you're going to suggest fighting a war for economic gain, at least try to pick one that would actually do. Otherwise, you're being evil and stupid.

And now for something completely different...

I'll be taking a break from p-values and peer effects tomorrow to watch the latest instalment of Sherlock now airing on Masterpiece Mystery. Mark Gatiss and Steven Moffat provide the scripts. That alone should be enough to get you to tune in.

Now if we can just get them to air Jekyll.

The Volunteer Effect

Before I jump back into the weeds here, let's step away for a paragraph and remind ourselves that the question of the moment is "Do charter schools provide a good, workable model for the overall education system?" This is a high bar, but it's one that the reformers themselves set. There are things that many charter schools do which tend to improve student performance and attitude, but which would lose their effectiveness if applied to the general population.

Which brings us to the volunteer effect.

In Influence, Robert Cialdini observes that "we accept inner responsibility for a behavior when we think we have chosen to perform it in the absence of strong outside pressures." (I couldn't find my copy but it's on page 97 of the library's.)* It's part of the chapter on commitment which is filled facts and examples that are relevant to the current education debate.

Indoctrination plays a large part in the culture of schools like KIPP and HCZ. This isn't a bad practice -- all schools do it to some extent -- but it is particularly important when a school tries to greatly increase student work load. If you are assigning hours of homework every night and study sessions every Saturday and you expect reasonable compliance, you are going to have to make the students believe that hard work is the right thing to do and that it will lead to large rewards.

That kind of belief modification is easiest when subjects see themselves as volunteers who not only chose to engage in the accompanying behaviors but who actually made an effort to do so. This works well in the current incarnation of charter schools, but everyone can't be a volunteer. When educators show the ability to inspire these beliefs in the kids who don't want to be there, then we'll have something.







* As part of this discussion, Cialdini also mentions that small rewards tend to create more long-lasting behavior changes than large rewards. This sets up an interesting topic for a future blog post on economics and compensation.

Fortune's smartest business books

I came across this while looking up some background for an upcoming post and clicked the link mostly of a sense of morbid curiosity (cynical people should generally avoid the business section of their neighborhood book store), but I was very pleasantly surprised.

Covers everything from Wealth of Nations to Blackhawk Down. Definitely worth a browse.

Mysteries of resonance

Messages are always a bit strange when you're not the target audience, but usually you can manage some level of understanding if you look at it from the target's perspective.



I don't agree with this protester's tee-shirt but I understand how he feels. On some level we all consider politicians in the other party liars and I suspect we have all indulged in the fantasy of saying some variation of "you lie" to a politician we particularly dislike and disagree with.

But sometimes, no matter how you try to see things from the other guy's vantage, you simply can't understand how something provokes its intended response.

Carly Fiorina's campaign spent tens of millions of dollars on an ad showing a clip of Barbara Boxer insisting that a witness at a Senate hearing refer to her as 'senator' saying "I worked so hard for that title." Now another group is using this phrase as a tagline.



Perhaps it would be different if I could have gotten Zucker's video to work, but I can't imagine how hearing a senator say "I worked so hard for that title" would generate much of an emotional response. Kay Bailey Hutchison worked hard to get into the Senate. So did Orin Hatch. So did every senator I can think of.

Is there an emotional resonance I'm missing here? Or is this a miscalculation on the part of Fiorina and company (keep in mind, these are the same people who thought a Bond villain would make a good spokesman).

The most open-label of open-label experiments

A big part of being a statistician is knowing when to get nervous, knowing when cutting a small corner could produce massive bleeding. One of those cut corners we pay particularly close attention to is the open-label trial, where the subjects know exactly what treatments they are getting.

In a perfect world, neither the subjects in a trial or the people administering the treatments would ever know who was getting what. This double-blind approach protects us from the placebo effect, which has an unfortunate way of popping up whenever humans are the subjects of research.

At the risk of being obvious, it is next to impossible to perform a double-blind experiment in most areas of educational research -- everyone knows who got the treatment and who didn't -- but that doesn't mean that the underlying reasons for preferring double-blind tests aren't there. We routinely allow for the possibility that the placebo effect can affect pretty much everything from surgery to the immune system. Does ignoring the possibility that it might affect student performance seem like a safe assumption?

If anything, education is a textbook example of an area where we would prefer not to use an open-label approach. We are working with a test population that's highly suggestible and treatments that rely heavily on the subjects' attitude. Under these conditions, telling subjects that they are about to receive a treatment is highly likely to bias the results.

And in the case of charter schools, students aren't just told they are about to receive a treatment; they are often told, in the most dramatic way possible, that they are about to have their lives transformed. Pedagogically, this is a good idea. It helps establish the belief that the student will succeed, a belief that can easily become self-fulfilling. Statistically, though, it greatly muddies the waters.

Watch the first minute of this Sixty Minutes segment. Look at the expressions of the father leaping out of his seat with excitement and the mother and daughter crying with joy. You'd have to be emotionally dead not to empathize with this family's feelings, but you'd also have to be a poor statistician not to wonder if those emotions contributed to student success.



update: I probably should have mentioned that there is reason to suspect that group dynamics may amplify the placebo effect here.

Implications of "counseling out" part II -- missions

This is the second part of a reply to a previous comment about counseling out disruptive students. In the first part I discussed how charter schools' greater freedom to get rid of problem students makes reliable public-to-charter school comparisons difficult, but as important as that question is, there's another that's even more fundamental when discussing these practices: what exactly do we want charter schools to do?

We have two basic choices:

The first is that charter are what all schools should eventually be. In this model, charters provide the template for the American education system. If this is what we're asking from them, then charters' problems dealing with disruptive and non-cooperative students is a serious failure;

If, however, we look at charter schools as niche programs designed to target specific areas and subpopulations, then counseling out student for academic or behavioral reasons may not be a problem at all. If the purpose of these schools is to allow room for experimentation, pump additional resources into under-served areas and provide a better match for certain kids who aren't getting what they should from the one-size-fits-all approach, then counseling out is a necessary part of the model.

Most champions of charter schools would probably pick the first model but most of the major criticisms that have been made recently about charters schools (data biasing issues, accusations of cherry-picking, questions about scalability) largely go away under the second model.

I have a feeling we'll be coming back to this one.

Implications of "counseling out" part I -- metrics

This comment from Michael Bish raises a couple of big questions:

One important point is that there is a big difference between counseling a student out because they have low test scores and counseling them out because they are actively disrupting the education of their peers.

The first issue here involves metrics. The idea that you should build a school system around a handful of test scores is one of the central tenets of the reform movement. Test scores are supposed to drive funding, contracts, allocation of resources, evaluations, bonuses, even terminations, but if you give certain schools more freedom to get rid of disruptive students the whole system breaks down.

Disruptive students not only take up time from the teacher that could be used in instruction; they also make it difficult for students to concentrate and tempt other others to act up. The result is that everyone's test scores go down.

By getting this student transferred you've not only raised the scores of an entire classroom of your students; you've lowered the scores of a comparable number of students in a public school in the same area. Since your school's evaluation, funding, and future contracts are dependent on your performance relative to other schools in the area, you can get a substantial double lift out of that single transfer.

And since public school to a large extent have to work with the students they are given, charter schools always win this one.