Not quite grasping the placebo concept

Marcus A. Winters writing in the Manhattan Institute's City Journal says:

The charters’ random-selection policy has allowed Hoxby’s team to evaluate them with a randomized field trial (RFT), the kind of experiment often described by social scientists as the “gold standard” in research design. RFTs resemble the clinical trials performed by pharmaceutical companies, which test a new drug by assembling a pool of subjects, randomly assigning subjects to receive either the drug or a placebo, and then comparing the subjects’ condition. Similarly, Hoxby took a pool of subjects (students applying to New York City charter schools); took advantage of the random nature of the lotteries, which assigned the subjects either to charters or to traditional public schools; and then compared their academic achievement. Because access to a charter school is the only meaningful difference between the two groups—both applied to charter schools, after all—comparing the groups’ later achievement really will tell us what effect charters have on student performance.

I've seen researchers and commentators ignore potential issues with open-label studies (see here and here for background), but Mr. Winters is the first not only to mention placebos but to compare them explicitly to public schools. At the risk of beating a man while he's down, the use of placebos is based on the assumption that knowing that you are being treated can and often does produce a "meaningful difference between the two groups."

Caroline Hoxby never mentioned placebos in this paper and I'm certain she would never suggest that being randomly assigned a public school is equivalent to receiving a placebo in a double blind test.

Unfortunately Dr. Hoxby is not the one writing the articles.

Not sure about this one...

John Judis is not someone you want to pick a fight with on the demographics of politics but I'm not sure he's made his case here (emphasis added):

The other telltale sign of Obama’s failure was the youth vote. Democratic victories in 2006 and 2008 very much depended upon increased support from and turnout among young voters. In 2008, Obama’s organization specifically targeted these voters. In this election, voters 18 to 29 again favored Democrats by a whopping 56 to 40 percent in House races. But they constituted only 11 percent of the electorate this year compared to 18 percent in 2008 House races and 12.5 percent in 2006. Obama and his political aides recognized that this was a problem, and in the last weeks of the election, tried to rouse these voters (hence all those campus rallies and the “Daily Show” appearance). But it was too late.

The natural comparison here seems to be with 2006 and the drop doesn't seem that big, particularly given anti-war issue (something that, even without the draft, would tend to skew by age) four years ago.

Any thoughts?

More on Health Care Efficiency

An update here on medical insurance with an inventory problem.

Depending on the condition being discussed, this can be a pretty serious matter and remedies are unclear. At the very least we are seeing the following as the solution:

1) Make a visit to an MD to get a prescription (cost of MD)

2) Visit a pharmacy and argue about insurance coverage (cost of administration)

3) Take much of a day off work (this is a cost to the employer via sick days or employee via needing to work longer hours to make up for this)

If these issues are frequent then the inventory control system is imposing a lot of costs on the system. Now, if this was a store (say Amazon) then it would be fighting to remedy these issues before it went bankrupt. But the lack of customer mobility is another area where Health Care is an unusual market.

"How much [math, literature, history, politics and music] do we really need?"

Last night I posted a long excerpt from G. V. Ramanathan's op-ed in the Washington Post. Though he made some valid points, I couldn't fully endorse the essay because of paragraphs like this:

Unlike literature, history, politics and music, math has little relevance to everyday life. That courses such as "Quantitative Reasoning" improve critical thinking is an unsubstantiated myth. All the mathematics one needs in real life can be learned in early years without much fuss. Most adults have no contact with math at work, nor do they curl up with an algebra book for relaxation.

There are two ways of looking at this, but whichever you pick, this analogy doesn't hold up.

First, we can view that what we learn as transferable. This view is consistent with my experience. I went from a BFA in creative writing to a master's in statistics and found that reading and writing about Shakespeare, Twain and Faulkner was good preparation for classes like experimental design and non-parametric methods. I have also found the reverse true: training as a statistician makes you a better critic.**

If problem solving, pattern recognition, logical thinking and other related skills can be transferred to a non-mathematical context then we can make an excellent for teaching more math (though probably not for the way we've been teaching it). If these skills aren't transferable, we can still make a case for math based on statistics and on spreadsheets (everyone has to deal with statistics as a citizen and consumer and every business I've seen could use at least one person who was good on Excel). We can no longer, however, make any kind of case for literature, history or music.

Let's start with literature. It is true that many people read books for enjoyment, but how much effect do literature classes have on what books people read after graduation? The adult recreational reading experience is largely independent of anything learned in school past eighth grade (working under the assumption that a proficient eighth grade reader should be able to handle most of the books people read for pleasure).

We keep teaching literature through high school and include it in the general ed requirements for college because we believe that what students learn in those classes is, to some degree, transferable. If you honestly believe that reading Macbeth does nothing but make you better at reading Elizabethan dialogue, there is no way to justify the waste of class time covering it.

The situation for history is even worse. You can at least make a reading-and-writing case for literature; barring the writing of erudite-sounding op-eds and blog posts, when's the last time you actually needed a historical fact? A few do occasionally come in handy for understanding current events, but the overwhelming majority have no conceivable value unless we assume that students are capable of transferring what they've learned about the long dead to questions regarding the living.

The case for music and art are strictly based on enrichment.

As for politics, I can't rebut this because I'm not sure what he's talking about. Other than a year of junior high civics, we don't really teach politics. We teach courses that are relevant to politics like history, economics and statistics, but the first one requires the students to generalize what they've learned and the last two require lots of you know what.

Professor Ramanathan makes some great points about the marketing of mathematics (read the whole thing) and I'm glad to see national attention brought to the question of what we should be teaching, but it's a complex and subtle question and the professor's Gordian approach isn't going to cut it.



* Along these lines, I did a couple of posts on economics and popular literature here and here and and a post on economics and film criticism here.

Ask your doctor if Sucrosa might be right for you

Unfortunately, this is just the sort of thing that won't be covered under Obamacare, so make sure to stock up.

FDA Approves Sale Of Prescription Placebo

September 17, 2003 | ISSUE 39•36

WASHINGTON, DC—After more than four decades of testing in tandem with other drugs, placebo gained approval for prescription use from the Food and Drug Administration Monday.

"For years, scientists have been aware of the effectiveness of placebo in treating a surprisingly wide range of conditions," said Dr. Jonathan Bergen of the FDA's Center for Drug Evaluation and Research. "It was time to provide doctors with this often highly effective option."

In its most common form, placebo is a white, crystalline substance of a sandy consistency, obtained from the evaporated juice of the Saccharum officinarum plant. The FDA has approved placebo in doses ranging from 1 to 40,000 milligrams.

...

Pain-sufferers like Margerite Kohler, who participated in a Sucrosa study in March, welcomed the FDA's approval.

"For years, I battled with strange headaches that surfaced during times of stress," Kohler said. "Doctors repeatedly turned me away empty-handed, or suggested that I try an over-the-counter pain reliever—as if that would be strong enough. Finally, I heard about Sucrosa. They said, 'This will work,' and it worked. The headaches are gone."

Researchers diagnosed Kohler with Random Occasional Nonspecific Pain and Discomfort Disorder (RONPDD), a minor but surprisingly pervasive medical condition that strikes otherwise healthy adults.

RONPDD is only one of many disorders for which placebo has proven effective, Bergen said.
...

"All placebos are not the same," Eli Lilly spokesman Giles French said. "Pacifex is the only placebo that's green and shaped like a triangle. Pacifex: A doctor gave it to you."

Despite such ringing endorsements, some members of the medical community have spoken out against placebo's approval, saying that the drug's wide range of side effects is a cause for concern.

"Yes, placebo has benefits, but studies link it to a hundred different side effects, from lower-back pain to erectile dysfunction to nausea," drug researcher Patrick Wheeler said. "Placebo wreaked havoc all over the body, with no rhyme or reason. Basically, whichever side effects were included on the questionnaire, we found in research subjects."

Added Wheeler: "We must not introduce placebo to the public until we pinpoint exactly how and why it works. The drug never should have advanced beyond the stage of animal testing, which, for some reason, was totally ineffective in determining its effectiveness."

In spite of the confusing data, drug makers say placebo is safe.

"The only side effect consistent in all test subjects was a negligible one—an almost imperceptible elevation in blood-glucose levels," French said. "It's unfair to the American people to withhold a drug so many of them desperately think they need."

Health Care

We haven't talked as much about Health Care lately, because I have been slammed by my first year of teaching. But I did want to highlight this post by Aaron Carroll:

You ready for this? They didn’t ship the medication. Why? It’s on back-order.

So explain this to me, awesome insurance system. You will only allow me to get my medication from one pharmacy, which is not local, and which you own and run. And you won’t send the medication early, which would provide a safe buffer. And then you run out of the medication. And then you don’t tell me until the day I run out. And then you don’t really tell me, you leave me a cryptic message on my voice mail that I would usually not be able to get until I arrive home and you are closed.

Will you allow me to go to another pharmacy? Sure. But you won’t cover it. Will you have the medication soon? Yeah, you hope to have it tomorrow.

I don’t believe you.

Do you have any concern – whatsoever – that I am without my medication for a period of time? That it’s entirely your fault?

I’m a model patient. I pay all my bills. I go to the doctor. I get the labs done. I refill the meds on time with a weeks’ advance. I follow the rules. And you screw me.

Are we really sure that the health care system in the United States meets the requirements for market efficiency*? Really?


* And, if you have not read the classic post by Mark Thoma then now is as good a time as any. Well worth the ten minutes to breeze through it and it is a very good way of evaluating whether a particular market is likely to be efficient.

More thoughts on Charter Schools

Mark quotes Diane Ravitch about the new film Waiting for Superman.

I think that these paragraphs really deserve some more discussion:

Geoffrey Canada is justly celebrated for the creation of the Harlem Children’s Zone, which not only runs two charter schools but surrounds children and their families with a broad array of social and medical services. Canada has a board of wealthy philanthropists and a very successful fund-raising apparatus. With assets of more than $200 million, his organization has no shortage of funds. Canada himself is currently paid $400,000 annually. For Guggenheim to praise Canada while also claiming that public schools don’t need any more money is bizarre. Canada’s charter schools get better results than nearby public schools serving impoverished students. If all inner-city schools had the same resources as his, they might get the same good results.

But contrary to the myth that Guggenheim propounds about “amazing results,” even Geoffrey Canada’s schools have many students who are not proficient. On the 2010 state tests, 60 percent of the fourth-grade students in one of his charter schools were not proficient in reading, nor were 50 percent in the other. It should be noted—and Guggenheim didn’t note it—that Canada kicked out his entire first class of middle school students when they didn’t get good enough test scores to satisfy his board of trustees

I note that the Harlem's Children's Zone seems to get only positive press in the blogosphere. Yet two facts here really seem to stand out. One, making the compensation structure that of a corporation is going to mean that we get advertising and not honest accounting of performance. A very high salary ($400,000 per year) and a corporation that lives or dies on profit does not incent actors to give a balanced view of the shortcomings of the corporation.

The second, and more damning, issue is that kicking out an entire class of students when their tests scores are low is guaranteed to ensure that Charter schools look better -- after all, the weak students have been ruthlessly removed. Anybody can look competent under this system.

This cuts to the heart of my discomfort with the charter school system. One of the most important issues facing American education is the drop-out of vulnerable students. Charter schools that are willing to remove poorly preforming students for reasons of "optics" cannot, in aggregate, improve this situation.

[Mark has already extensively documented other concerns with interpreting charter school performance]

Nervous Nate -- why timidity at FiveThirtyEight is a sign of professionalism

I've been working on an article on educational statistics. One of the major themes is the way statisticians should handle research when they don't have much faith in the results. This is not quite the same as the way Bayesians assign weights to beliefs but it's not all that different. Us poor, simple frequentist-folk approach the problem in an informal way and try to present our results with sufficient caveats that the listeners will understand that we may have a small p-value but that doesn't mean we'd bet the farm.

As I was working on that part of the article, I was struck by how Nate Silver was doing exactly what I was describing, pointing out the factors that could throw his forecasts off, reminding people that in this election, more than most, anything could happen.

That's one of the reason's Silver is the gold standard* in the field.





* Sorry, I just couldn't help myself.

An excellent Maxine Udall post

Really good stuff (via guess who):

In 1940, Howard Florey and Ernst Chain, at Oxford University, produced enough stable chemical from a mold, penicillium, to test on mice infected with streptococcus. Of eight infected, 4 were treated and recovered rapidly. "Eureka!," you say. "The rest is history, yes?"

Well, no. Not quite.

Florey and a colleague packed up their penicillin and their results and took them to major pharmaceutical companies in the UK, the US, and Canada. None wanted to invest the capital in scaling it for production, mainly because there were signs that bacteria were becoming resistant to the sulfa drugs they were already manufacturing. Even if penicillin were effective for a while, it would eventually become ineffective and demand would dwindle. Where's the profit in that, they rightly asked?

So how did we the people get penicillin (and the many subsequent antibiotics that repeatedly haul us back from the brink of death, bronchitis, and septicemia)? The US government stepped in. The US military pushed a bit on this. After all, who better to sense on the eve of war the potential gains from a new antibiotic? Roughly half of all deaths in at least one previous war had been from infection.

It was a group of government scientists at the US Department of Agriculture who scaled production and increased the drug's efficacy four-fold. And in record time. Imagine. Guvmint scientists produced something quickly, efficiently, and made it (and us) better.

I know what you're thinking. You're thinking that surely now, after the US government and the US taxpayers had volunteered the start up costs, the drugs manufacturers must have agreed to produce it, yes? By the time it had been scaled, it was more apparent than ever that the US might be drawn into a major world war. The military wanted it. Surely an appeal to patriotic duty would move them?

No. I'm afraid not. Not until George Merck, CEO of Merck & Company, agreed to do it and persuaded the heads of several other manufacturers to join him. "Medicine is for people, not for profits," George said.

"How much math do we really need?"

I disagree with a fair chunk of this, but it's worth reading:

Twenty-seven years have passed since the publication of the report "A Nation at Risk," which warned of dire consequences if we did not reform our educational system. This report, not unlike the Sputnik scare of the 1950s, offered tremendous opportunities to universities and colleges to create and sell mathematics education programs.

Unfortunately, the marketing of math has become similar to the marketing of creams to whiten teeth, gels to grow hair and regimens to build a beautiful body.

There are three steps to this kind of aggressive marketing. The first is to convince people that white teeth, a full head of hair and a sculpted physique are essential to a good life. The second is to embarrass those who do not possess them. The third is to make people think that, since a good life is their right, they must buy these products.

So it is with math education. A lot of effort and money has been spent to make mathematics seem essential to everybody's daily life. There are even calculus textbooks showing how to calculate -- I am not making this up and in fact I taught from such a book -- the rate at which the fluid level in a martini glass will go down, assuming, of course, that one sips differentiably. Elementary math books have to be stuffed with such contrived applications; otherwise they won't be published.

You can see attempts at embarrassing the public in popular books written by mathematicians bemoaning the innumeracy of common folk and how it is supposed to be costing billions; books about how mathematicians have a more clever way of reading the newspaper than the masses; and studies purportedly showing how much dumber our kids are than those in Europe and Asia.

As for the third, even people who used to proudly proclaim their mathematical innocence do not wish to abridge the rights of their children to a good life. They now participate in family math and send the kids to math camps, convinced that the path to good citizenship is through math.

We need to ask two questions. First, how effective are these educational creams and gels? With generous government grants over the past 25 years, countless courses and conferences have been invented and books written on how to teach teachers to teach. But where is the evidence that these efforts have helped students? A 2008 review by the Education Department found that the nation is at "greater risk now" than it was in 1983, and the National Assessment of Educational Progress math scores for 17-year-olds have remained stagnant since the 1980s.

The second question is more fundamental: How much math do you really need in everyday life? Ask yourself that -- and also the next 10 people you meet, say, your plumber, your lawyer, your grocer, your mechanic, your physician or even a math teacher.

Unlike literature, history, politics and music, math has little relevance to everyday life. That courses such as "Quantitative Reasoning" improve critical thinking is an unsubstantiated myth. All the mathematics one needs in real life can be learned in early years without much fuss. Most adults have no contact with math at work, nor do they curl up with an algebra book for relaxation.

Diane Ravitch vs. "Superman"

There are few places where the divide in the educational debate between popular perception reality is as depressing as the reaction to Waiting for Superman. Here Diane Ravitch (whose resume includes the first Bush administration and the Hoover Institute) lists some of the distortions, omissions and factual errors.

More on this later.

"The Freakonomics Fallacy"

From an excellent post by Barbara Kiviat (pay close attention to the last sentence):

There’s been some interesting discussion in response to my earlier post about why we expect too much from economists, although a lot of the comments miss my larger point. What I was trying to say is that economics might not entirely be up to the task of explaining what we generally consider to be economic phenomena because we are overconfident about what the discipline has the ability to account for. You might call this the Freakonomics Fallacy. Whatever it is in the world we are trying to explain—crime, climate change, test scores—economics has the answer.

If this isn’t in fact true, then why would we think it? Again, to underscore a part of my earlier argument that I probably didn’t make forcefully enough: we think economics has all the answers because economics has become our major mode of understanding the social world around us. Charities are social businesses. Policy makers are cost-benefit analyzers. Education is a market.

This has not always been the case. In earlier eras, we were often more likely to understand human behavior and social dynamics through other prisms, such as political science, sociology, psychology, anthropology or biology. Indeed, for better or for worse, many of these fields took a turn being the dominant social science. I’m not saying that one frame gives a more accurate or useful picture of the world than another. Just that each leads to a different way of understanding why things happen the way they do because each comes with its own set of assumptions and simplifications.

You can see my thoughts on the subject here.

Children and Placebos

From:

Greater Response to Placebo in Children Than in Adults: A Systematic Review and Meta-Analysis in Drug-Resistant Partial Epilepsy

(published in PLoS Medicine)

The researchers searched the literature for reports of RCTs on the effects of antiepileptic drugs in the add-on treatment of drug-resistant partial epilepsy in children and in adults—that is, trials that compared the effects of giving an additional antiepileptic drug with those of giving a placebo by asking what fraction of patients given each treatment had a 50% reduction in seizure frequency during the treatment period compared to a baseline period (the “50% responder rate”). This “systematic review” yielded 32 RCTs, including five pediatric RCTs. The researchers then compared the treatment effect (the ratio of the 50% responder rate in the treatment arm to the placebo arm) in the two age groups using a statistical approach called “meta-analysis” to pool the results of these studies. The treatment effect, they report, was significantly lower in children than in adults. Further analysis indicated that this difference was because more children than adults responded to the placebo. Nearly 1 in 5 children had a 50% reduction in seizure rate when given a placebo compared to only 1 in 10 adults. About a third of both children and adults had a 50% reduction in seizure rate when given antiepileptic drugs.

...

These findings, although limited by the small number of pediatric trials done so far, suggest that children with drug-resistant partial epilepsy respond more strongly in RCTs to placebo than adults. Although additional studies need to be done to find an explanation for this observation and to discover whether anything similar occurs in other conditions, this difference between children and adults should be taken into account in the design of future pediatric trials on the effects of antiepileptic drugs, and possibly drugs for other conditions. Specifically, to reduce the risk of false-negative results, this finding suggests that it might be necessary to increase the size of future pediatric trials to ensure that the trials have enough power to discover effects of the drugs tested, if they exist.

Of course, with open-label experiments, this should lead to false positives.

And a final word on the subject from America's most successful philosophy major

But I quit that! I've quit ALL drugs. Well... let me say one thing: I twisted my ankle this morning, and I was in quite a bit of pain... so I went to the doctor, and I asked him to give me some pain pills. And he didn't want to do it, but I talked him into it. So he gave me some pills -- and I shouldn't have done this, but I took some about an hour before the show tonight, and right now... I am high... as a KITE! I mean, it is unbelievable! And I would NEVER say this to you people, but, in this case: if you EVER get a chance, to take these drugs... DO IT! They're called... [ he glances from side-to-side cautiously ] Placebos! I mean, I'm thinking that right now I have NO idea where I am at all! It is WILD! [in a hushed voice] PLUH-SEE-BO!

Steve Martin on SNL

Placebos

This is relevant to Mark's posts on how comparisons between students who win and lose charter school lotteries are similar to open label drug trials. Andrew Gelman quotes Kaiser Fung stating that (in drug trials) the standard comparison is:

Effect on treatment group = Effect of the drug + effect of belief in being treated
Effect on placebo group = Effect of belief in being treated

I think that the mathematical formulation of Mark's concern about using charter school lotteries as follows:

Effect in charter school group = Effect of the charter school intervention + effect of belief in being treated + baseline

Effect on students in standard schools = baseline - (any discouragement effect from losing the lottery)

So if the effect in the charter school students is greater than that in students who lose the lottery and are placed in standard schools, that skill doesn't separate the two sources of variation (placebo versus innate effect of the intervention).

The real test would be to introduce some sort of placebo intervention (a charter school that used standard educational approaches??). But that is not an easy thing to accomplish, for obvious reasons. I suspect that this is why psychologists end up having to use deception as part of their studies, despite the ethical quagmires this produces.

UPDATE: missing link fixed