Homework and treatment levels

Previously on OE, we've talked about how norming, placebo and volunteer effects can cause methods like lottery-based analyses to miss significant sampling effects. Here's one more area of potential concern.

The basic problem here is familiar to clinical researchers. You're trying to determine the optimal level of a drug or therapy or exercise regime or type of food. There is an ideal dosage, the level of treatment you would prescribe if your only concern was optimizing the effect. Then there is what we might call a realistic dosage, one that takes into account factors like side effects and compliance. These levels vary from group to group which means the process of finding the right dosage is sensitive to sampling issues. The appropriate level is likely to be different for a group of college students and a group of senior citizens.

Teachers assigning homework face essentially the same problems. Assuming that goal is to improve students' score on a standardized test, there is an ideal homework assignment for each lesson, a subset of the problems available in the text that will tend to maximize the average score of a class (to keep things simple we'll limit the discussion, with no loss of generality, to homework assignments derived from the textbook).

The ideal assignment assumes that students have unlimited time and will complete all the assigned problems. The realistic assignment would take into account factors like time constraints, demands from other classes, compliance, burnout and parental pushback (I can tell you from experience that parents complain if they feel their children are being given excessive work and I would argue this is a good thing -- teachers should remember that their students' time also has value). The realistic assignment would optimize the class's average score when these constraints are in place.

Like the previously discussed norming, placebo and volunteer effects, a sampling/treatment level interaction with the homework assignment here can interfere with the ability of lottery-based analysis to detect sampling effects.

To see how this would work, consider a charter school like one in the KIPP system that very publicly acknowledges that students will be expected to do large amounts of homework each night (some schools even require parents to sign contracts agreeing to sign off on the students' homework). Students who apply for this school are aware of these requirements, as are their parents. This certainly raises the possibility that the optimal realistic homework level might be higher for these students, particularly if this is one of those schools that aggressively culls out the students who can't handle the workload.

If this is the case and if both the charter school and the public schools assign the optimal levels of homework for every class, you will have a sample based effect that will completely evade detection by a lottery-based analysis. That analysis will compare the performance of those who applied for the charter and were accepted against those who applied but lost the lottery. Since the rejected students will receive a treatment level that was optimized for the general population while the accepted students will receive a treatment level optimized to their particular subgroup, we expect the charter school students to do better, leading the lottery-based analysis to incorrectly conclude that there is no selection effect.

I used homework in this post to keep things simple but the principle applies just as well to most elements of the educational models of many highly praised charter schools -- longer days, Saturday instruction, extended school years, holding students responsible for more material on tests. Put bluntly, these schools get their results by giving students lots of work. Because of these results, these schools are often held up as examples for the rest of the educational system, but when you start with a self-selected sample then counsel out the students in that sample who have trouble keeping up, you should at least consider the possibility that the optimal workload for your student body is higher than the optimal workload for the general populace.

This doesn't mean that all high performing charter schools are simply running on undetected selection effects and it certainly doesn't mean that charter schools with high standards should lower them. What is it does mean is that selection effects in education show up in subtle and complex ways. It remains extraordinarily difficult to resolve these issues through observational means. Any study that claims to have settled them should be approached with great caution.

Things you can do if you're a charter school

Make demands of parents:

Parent involvement has been a surprisingly central feature of many of the proposals for new and restructured charter schools in California. State legislation mandates that parents be involved in the governance of charter schools, but broader kinds of parental involvement seem to be contemplated by those who are designing these new schools. In many of these schools, parent involvement is much more than simply a requirement to volunteer assistance or to help with their children's homework. Parents are seen as central adults in a more inclusive school community--participants who share time and expertise with the school's students as a whole. For example, a majority of the first chartered schools were planning to have parents and other community members as instructors in the school building, and several expected to sponsor training in tutoring methods and parenting techniques for use at home. In fact, a survey of thirty-four of the first forty-four schools chartered by the state found that in more than 50 percent parents are required to sign contracts and to participate in certain activities (Dianda & Corwin, 1994). One recently approved charter school, for example, intends to require parents "to volunteer a minimum of three hours per month at the school." Another stated in its charter: "Parents, by signing their child's registration form, commit themselves to at least 2 hours of school service per month.... Any student accepted on an above mentioned agreement will meet a prescribed written contract and will understand, if the contract is broken, said agreement will be revoked and the student will be disenrolled."

In many ways this is a good thing and I'm not saying that charter schools should stop, but it does provide another example of the near impossibility of making meaningful comparisons between schools that have the option of easily 'disenrolling' students and those that don't.

The most open-label of open-label experiments

A big part of being a statistician is knowing when to get nervous, knowing when cutting a small corner could produce massive bleeding. One of those cut corners we pay particularly close attention to is the open-label trial, where the subjects know exactly what treatments they are getting.

In a perfect world, neither the subjects in a trial or the people administering the treatments would ever know who was getting what. This double-blind approach protects us from the placebo effect, which has an unfortunate way of popping up whenever humans are the subjects of research.

At the risk of being obvious, it is next to impossible to perform a double-blind experiment in most areas of educational research -- everyone knows who got the treatment and who didn't -- but that doesn't mean that the underlying reasons for preferring double-blind tests aren't there. We routinely allow for the possibility that the placebo effect can affect pretty much everything from surgery to the immune system. Does ignoring the possibility that it might affect student performance seem like a safe assumption?

If anything, education is a textbook example of an area where we would prefer not to use an open-label approach. We are working with a test population that's highly suggestible and treatments that rely heavily on the subjects' attitude. Under these conditions, telling subjects that they are about to receive a treatment is highly likely to bias the results.

And in the case of charter schools, students aren't just told they are about to receive a treatment; they are often told, in the most dramatic way possible, that they are about to have their lives transformed. Pedagogically, this is a good idea. It helps establish the belief that the student will succeed, a belief that can easily become self-fulfilling. Statistically, though, it greatly muddies the waters.

Watch the first minute of this Sixty Minutes segment. Look at the expressions of the father leaping out of his seat with excitement and the mother and daughter crying with joy. You'd have to be emotionally dead not to empathize with this family's feelings, but you'd also have to be a poor statistician not to wonder if those emotions contributed to student success.



update: I probably should have mentioned that there is reason to suspect that group dynamics may amplify the placebo effect here.

Implications of "counseling out" part II -- missions

This is the second part of a reply to a previous comment about counseling out disruptive students. In the first part I discussed how charter schools' greater freedom to get rid of problem students makes reliable public-to-charter school comparisons difficult, but as important as that question is, there's another that's even more fundamental when discussing these practices: what exactly do we want charter schools to do?

We have two basic choices:

The first is that charter are what all schools should eventually be. In this model, charters provide the template for the American education system. If this is what we're asking from them, then charters' problems dealing with disruptive and non-cooperative students is a serious failure;

If, however, we look at charter schools as niche programs designed to target specific areas and subpopulations, then counseling out student for academic or behavioral reasons may not be a problem at all. If the purpose of these schools is to allow room for experimentation, pump additional resources into under-served areas and provide a better match for certain kids who aren't getting what they should from the one-size-fits-all approach, then counseling out is a necessary part of the model.

Most champions of charter schools would probably pick the first model but most of the major criticisms that have been made recently about charters schools (data biasing issues, accusations of cherry-picking, questions about scalability) largely go away under the second model.

I have a feeling we'll be coming back to this one.

Implications of "counseling out" part I -- metrics

This comment from Michael Bish raises a couple of big questions:

One important point is that there is a big difference between counseling a student out because they have low test scores and counseling them out because they are actively disrupting the education of their peers.

The first issue here involves metrics. The idea that you should build a school system around a handful of test scores is one of the central tenets of the reform movement. Test scores are supposed to drive funding, contracts, allocation of resources, evaluations, bonuses, even terminations, but if you give certain schools more freedom to get rid of disruptive students the whole system breaks down.

Disruptive students not only take up time from the teacher that could be used in instruction; they also make it difficult for students to concentrate and tempt other others to act up. The result is that everyone's test scores go down.

By getting this student transferred you've not only raised the scores of an entire classroom of your students; you've lowered the scores of a comparable number of students in a public school in the same area. Since your school's evaluation, funding, and future contracts are dependent on your performance relative to other schools in the area, you can get a substantial double lift out of that single transfer.

And since public school to a large extent have to work with the students they are given, charter schools always win this one.

"Counseling out"

(This post also appears at Education and Statistics.)

Paul Tough writing in Slate recounts the following:

In Whatever It Takes, in one of the chapters on the Promise Academy middle school, I describe the impact of the KIPP schools in the Bronx and Harlem on the Promise Academy’s leaders and staff. This was during the first few years of the Harlem Children Zone’s middle school, which were a struggle, and those KIPP schools, which had very good test results, were for the Promise Academy administrators both a standard to be aspired to and a frustrating reminder that their own students weren’t performing at the same high level as KIPP’s students.

Terri Grey, the Promise Academy principal at the time, believed the attrition issue was part of what was holding her school back. As she put it to me in one conversation, “At most charter schools, if the school is not a good fit for their child, the school finds a way to counsel parents out”—to firmly suggest, in other words, that their child might be happier elsewhere. “Whereas Promise Academy is taking the most disengaged families and students and saying, ‘No, we want you, and we’re trying to keep you here, and we don’t want to counsel you out.” That policy made it impossible, she believed, for the Promise Academy to achieve KIPP-like results.

I’m not entirely convinced that that was the real problem at Promise Academy—or that the KIPP schools in New York were actually “counseling out” a significant number of students. But I do think it’s true that Geoffrey Canada’s guiding ethic has always been to go out of his way to attract and retain the most troubled parents and students. And that makes running a school, or any program, more difficult, even if it makes the mission purer and, in the end, more important.

For reasons I'll get to later, I suspect that the number of students you have to "counsel out" to have a significant effect on a school's test scores is lower than Mr. Tough realizes, but there are a couple of more important points.

The first is that selective attrition is recognized as a serious issue not just by critics of the reform movement but by responsible people within the charter school community.

The second is that all charter schools and charter school administrators are not interchangeable. There are some gifted educators with great ideas in that system. We've spent almost two decades overlooking the flaws in charter schools. It would be a serious mistake to try to compensate by overlooking the strengths.

"they are purging nonperforming students at an alarming rate"

(This post also appears at Education and Statistics.)

Mike at ScienceBlogs has some thoughts about selection by attrition:

A letter to Diane Ravitch from a Los Angeles school prinicipal documents just how dishonest and harmful this practice is (italics [Mike's]):

I received an email from Dr. DeWayne Davis, the principal of Audubon Middle School in Los Angeles, which was sent to several public officials. Dr. Davis said that local charter schools were sending their low-performing students to his school in the middle of the year. He wrote:

"Since school began, we enrolled 159 new students (grades 7 and 8). Of the 159 new students, 147 of them are far below basic (FBB)!!! Of the 147 students who are FBB, 142 are from charter schools. It is ridiculous that they can pick and choose kids and pretend that they are raising scores when, in fact, they are purging nonperforming students at an alarming rate--that is how they are raising their scores, not by improving the performance of students. Such a large number of FBB students will handicap the growth that the Audubon staff initiated this year, and further, will negatively impact the school's overall scores as we continue to receive a recurring tide of low-performing students."

Ravitch concludes:

Doing better than an under-resourced neighborhood school is not the same as getting "amazing results." Very few charters do. Probably less than 5 percent. Charters are not a silver bullet. They are a lead bullet. Their target is American public education.

This is just par for the course for modern conservatism: have private systems skim the cream, and leave the public sector to clean up an impossible mess. When they can't, this supposedly shows the inability of government to solve problems.

I have a few points to add:

1. This is a brutal way to treat these kids. You build their hopes up, then crush them, then dump the kids in a new school in the middle of the year;

2. We are talking about getting an influx of students who are badly behind and who are ready to give up and/or act out. This will disrupt classes slightly less than having a nearby car alarm go off at random times once or twice an hour;

3. But I think Ravitch overstates the case against charter schools. I've dealt with some small, independent schools that have impressed the hell out of me and I can see them playing an important role in our system, though a radically different role than Arne Duncan sees.

Perhaps this is the time for a counter-reformation

Just to review where we stand.


Charter schools

But for all their support and cultural cachet, the majority of the 5,000 or so charter schools nationwide appear to be no better, and in many cases worse, than local public schools when measured by achievement on standardized tests, according to experts citing years of research. Last year one of the most comprehensive studies, by researchers from Stanford University, found that fewer than one-fifth of charter schools nationally offered a better education than comparable local schools, almost half offered an equivalent education and more than a third, 37 percent, were “significantly worse.”

Although “charter schools have become a rallying cry for education reformers,” the report, by the Center for Research on Education Outcomes, warned, “this study reveals in unmistakable terms that, in the aggregate, charter students are not faring as well” as students in traditional schools.

(As I mentioned before, there is reason to believe that this research is biased in favor of charter schools.)


Test-based metrics

For a variety of reasons, analyses of VAM [Value Added Modeling] results have led researchers to doubt whether the methodology can accurately identify more and less effective teachers. VAM estimates have proven to be unstable across statistical models, years, and classes that teachers teach. One study found that across five large urban districts, among teachers who were ranked in the top 20% of effectiveness in the first year, fewer than a third were in that top group the next year, and another third moved all the way down to the bottom 40%. Another found that teachers’ effectiveness ratings in one year could only predict from 4% to 16% of the variation in such ratings in the following year. Thus, a teacher who appears to be very ineffective in one year might have a dramatically different result the following year. The same dramatic fluctuations were found for teachers ranked at the bottom in the first year of analysis. This runs counter to most people’s notions that the true quality of a teacher is likely to change very little over time and raises questions about whether what is measured is largely a “teacher effect” or the effect of a wide variety of other factors.

A study designed to test this question used VAM methods to assign effects to teachers after controlling for other factors, but applied the model backwards to see if credible results were obtained. Surprisingly, it found that students’ fifth grade teachers were good predictors of their fourth grade test scores. Inasmuch as a student’s later fifth grade teacher cannot possibly have influenced that student’s fourth grade performance, this curious result can only mean that VAM results are based on factors other than teachers’ actual effectiveness.

Firing under-performing teachers

If new laws or policies specifically require that teachers be fired if their students’ test scores do not rise by a certain amount, then more teachers might well be terminated than is now the case. But there is not strong evidence to indicate either that the departing teachers would actually be the weakest teachers, or that the departing teachers would be replaced by more effective ones.

Performance pay

The study was conducted by the National Center on Performance Incentives at Vanderbilt. The center, which takes no advocacy position on the issue, was created at the university's highly regarded Peabody College of Education and Human Development in 2006 with a $10 million federal research grant.

In a three-year experiment funded by the federal grant and aided by the Rand Corp., researchers tracked what happened in Nashville schools when math teachers in grades 5 through 8 were offered bonuses of $5,000, $10,000 and $15,000 for hitting annual test-score targets. About 300 teachers volunteered. Researchers randomly assigned half of the participants to a control group ineligible for the bonuses and the other half to an experimental group that could receive bonuses if their students reached certain benchmarks.

Researchers designed the bonuses to be large enough to function as a legitimate incentive for teachers whose average salary, according to a union official, is between $40,000 and $50,000. There were no additional variables in the experiment: no professional development, mentoring or other elements meant to affect test scores. The bonuses, totaling nearly $1.3 million, were funded by businessman Orrin Ingram, according to news reports. A university spokeswoman said Tuesday evening that she could not confirm those reports, and Ingram could not be reached for comment.

On the whole, researchers found no significant difference between the test results from classes led by teachers eligible for bonuses and those led by teachers who were ineligible. Bonuses appeared to have some positive effect in the fifth grade, researchers said, but they discounted that finding in part because the difference faded out when students moved to the sixth grade.

Just for the record, I believe that charter schools, increased use of metrics, merit pay and a streamlined process for dismissing bad teachers do have a place in education, but all of these things can more harm than good if badly implemented and, given the current state of the reform movement, badly implemented is pretty much the upper bound.

KIPP and the Teach(er) for America (or you don't know Jake)

Sometimes the most instructive passages are the most painful to read. This account from the New York Times had me wincing every third word (though I will admit to a little schadenfreude at seeing the top of her class at Stanford being outmaneuvered by Jake from Two and a Half Men).

Time permitting I'll do a post on why this was such a disastrous lesson and how a better, more experienced teacher would have done things differently. For now, though, let's approach it from the other side: what would happen if we kept Ms. Nguyen and lost Jake. Putting aside for a moment peer effects, assuming no one else steps up to take the role, how would the Jake-less class be different?

The lesson plan would still be weak, but this is a math class and most of the actual learning in a math class takes place after the lesson when the students start on their worksheets and homework. With more time and a less adversarial relationship with the class, the teacher can go from desk to desk, checking to make sure that problems are being done correctly and helping the students who are having trouble.

Now let's add in the peer effects. Jake has reset the norms of behavior for the class. He has also established that it is possible to jerk the teacher's chain and create great entertainment value with few negative consequences. In a Jake-less class this wouldn't be an issue. The inability to assert authority is only an issue when someone questions it.

In short, losing Jake should produce a substantial gain in student performance and classroom metrics.

Charter schools are designed to be Jake-free zones but none of the effects of removing Jake are likely to show up in the lottery-based analyses so favored by charter school supporters. This creates a fatally flawed set of metrics.

Worse yet, it creates a system of reforms that have, too often, based their claims of success on leaving behind the very students who needed the most help.

"We've made huge advances in what they're called" -- New Republic edition

This post by Joseph got me thinking. Charter schools are private contractors providing services that were previously provided by the government. Any statement that's true about charter schools should still be true if you substitute in the phrase "some private contractors."

But if you actually make the substitution, you often end up with statements the author would never think of making. Statements like this:

But Mead says ... she’s seen Gray hint that he’d like to more tightly regulate [private contractors]. “We have a law that gives a tremendous amount of autonomy to the [private contractors] but enables them to implement programs that can be effective. If you try to put more regulation on that, if can dissuade people from [privatizing],” Mead says.

Would Seyward Darby normally describe a push for tighter regulation of private contractors as "disappointing"? Would the New Republic normally endorse a candidate because he was against stricter regulation of private contractors? Would everyone take a moment and see if Rod Serling is taking a smoke break in the vicinity?

I strongly believe that there is a place for charter schools in our system, but those schools have to meet exactly the same criteria as other contractors. Two of those criteria are transparency and openness to regulation, and given recent history, it's safe to say that some charter schools are failing these tests.

[note: typo in the title has been corrected]

Strange Bedfellows

(where your intrepid blogger sides with Robert Samuelson, gives Ray Fisman a break and wonders what in the hell happened to Jonathan Chait)

You really have to read this Jonathan Chait column on Robert Samuelson, and I don't mean that in a good way. You have to read this to see for yourself how low the educational reform movement can drag even a writer as gifted as Chait.

There's too much here to cover in one post (I could do a page just on Chait's weird reaction to Samuelson's looks, a topic that I had never given any thought to up until now). I may take another pass at another section later but for now I'm going to limit myself to this particularly egregious bit:

How does Samuelson explain the existence of new charter schools that produce dramatically higher results among these lazy, no-good teenagers? He insists, "no one has yet discovered transformative changes in curriculum or pedagogy, especially for inner-city schools, that are (in business lingo) 'scalable.'" This is utterly false. The most prominent example is the Kipp schools, which have shown revolutionary improvements among poor, inner-city students and have rapidly expanded.

It is strange to see Chait take the pro-privatization side of the debate, stranger still to see him accuse critics of charter schools of having an anti-government bias*, but what pushes this into Rod Serling territory is the spectacle of having Chait, one of the most gifted bullshit detectors of the Twenty-first Century, rolling out the same sort of flawed argument that he has made a career out of dismantling.

In order to be viable, a reform has to improve on the existing system by a large enough margin to justify its implementation costs, but if you accept the metrics used by the reform movement, then you will have to conclude that charter schools do worse than public schools more often than they do better.**

So we have a major push to privatize government services which, after about two decades of testing have been shown to under-perform their traditional government-run alternatives. Rather than show why this statistic is misleading, Chait pulls out vague, anecdotal evidence of a single out-lier. Now, given the variability of the data, we would expect the top schools (or even chains) to do pretty well. That alone rebuts Chait's point, but it gets worse. Self-selection, peer effects and selective attrition*** all artificially inflate KIPP's results. When you take these factors into account, it's hard to make a compelling statistical case that even the best charter schools are outperforming public schools (though the second footnote still applies).

At the risk of over-emphasizing, this is Jonathan -- freaking -- Chait we're talking about, a writer known for his truly exceptional gift for constructing logical arguments and, more importantly, spotting the fallacies in the arguments of others. Under normal conditions, Chait would never fall for a badly presented argument-by-anomally, let alone make one, just as, under normal circumstances, a confrontation between Samuelson and Chait would result in little pieces of the former being scraped off of the walls of the Washington Post.

But Chait loses this confrontation decisively. From his ad hominem opening to his factually challenged close he fails to score a single point. And this is far from the only example of this odd reform-specific impairment affecting otherwise accomplished writers. OE has spilled endless pixels on the reform-related lapses, both statistical and rhetorical, of smart, serious, dedicated people like Chait, Seyward Darby and, of course, Ray Fisman (just do a keyword search). None of these people would normally produce the kind of work we've cataloged here. None of them would normally ignore the defection of one of the founding members of the reform movement. None of these people would normally feel comfortable dismissing without comment contradictory findings from EPI, Donald Rubin and the Rand Institute.

David Warsh has aptly made the following comparison:

Remember the recipe for a policy disaster? Start with a handful of policy intellectuals confronting a stubborn problem, in love with a Big Idea. Fold in a bunch of ambitious Ivy League kids who don’t speak the local language. Churn up enthusiasm for the program in the gullible national press – and get ready for a decade of really bad news. Take a look at David Halberstam’s Vietnam classic The Best and the Brightest, if you need to refresh your memory. Or just think back on the run-up to the war in Iraq.

but along with Halberstam, it might be time to brush off our copies of Cialdini's Influence.

From a data standpoint, the past few years have been rough on the reform movement. Charter schools have been shown to be more likely to under-perform than to outperform. Joel Klein's spectacular record turned out to be the product of creative accounting (New York City schools have actually done much worse than the rest of the state). Findings contradicting the fundamental tenets of the movement accumulated. Major figures in research (Rubin) and education (Ravitch) have publicly questioned the viability of proposed reforms.

As Cialdini lays out in great detail, when you challenge people's deeply held beliefs with convincing evidence, you usually get one of two responses. Sometimes you will actually manage to win them over. More often, though, they will dig in, embrace their beliefs more firmly and find new ways to justify them.

I think it's safe to say we don't have response number one.




* Almost all of the major tenets of the modern reform can be traced back to the Reagan era and were closely associated with the initiatives described in Franks' The Wrecking Crew.

** Ironically, if you consider the intellectual framework of the reform movement to be flawed and overly simplistic, you can actually make a much better case for charter schools.

*** From Wikipedia: "In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."

"Shape of the earth -- opinions still differ"

I was reminded of Paul Krugman's parody of a New York Times headline when I came to this NYT headline:

"Despite Push, Success at Charter Schools Is Mixed By TRIP GABRIEL"

Followed a few paragraphs later by the money shot:

But for all their support and cultural cachet, the majority of the 5,000 or so charter schools nationwide appear to be no better, and in many cases worse, than local public schools when measured by achievement on standardized tests, according to experts citing years of research. Last year one of the most comprehensive studies, by researchers from Stanford University, found that fewer than one-fifth of charter schools nationally offered a better education than comparable local schools, almost half offered an equivalent education and more than a third, 37 percent, were “significantly worse.”

Although “charter schools have become a rallying cry for education reformers,” the report, by the Center for Research on Education Outcomes, warned, “this study reveals in unmistakable terms that, in the aggregate, charter students are not faring as well” as students in traditional schools.

As I mentioned before, there is reason to believe that this research is biased in favor of charter schools.

If you showed me test results for a new cholesterol-controlling drug in which 20% of the subjects had lower LDL levels than when they started taking the drugs, 51% stayed the same and 37% were "significantly worse," I don't think I would describe the results as 'mixed.'

But, of course, I'm not writing for the New York Times.

Harlem Children's Zero Sum Game

I used to work in the marketing side of large corporation (I don't think they'd like me to use their name so let's just say you've heard of it and leave the matter at that). We frequently discussed the dangers of adverse selection: the possibility that a marketing campaign might bring in customers we didn't want, particularly those we couldn't legally refuse. We also spent a lot of time talking about how to maximize the ratio of perceived value to real value.

On a completely unrelated note, here's an interesting article from the New York Times:

Pressed by Charters, Public Schools Try Marketing
By JENNIFER MEDINA

Rafaela Espinal held her first poolside chat last summer, offering cheese, crackers and apple cider to draw people to hear her pitch.

She keeps a handful of brochures in her purse, and also gives a few to her daughter before she leaves for school each morning. She painted signs on the windows of her Chrysler minivan, turning it into a mobile advertisement.

It is all an effort to build awareness for her product, which is not new, but is in need of an image makeover: a public school in Harlem.

As charter schools have grown around the country, both in number and in popularity, public school principals like Ms. Espinal are being forced to compete for bodies or risk having their schools closed. So among their many challenges, some of these principals, who had never given much thought to attracting students, have been spending considerable time toiling over ways to market their schools. They are revamping school logos, encouraging students and teachers to wear T-shirts emblazoned with the new designs. They emphasize their after-school programs as an alternative to the extended days at many charter schools. A few have worked with professional marketing firms to create sophisticated Web sites and blogs.
...

For most schools, the marketing amounts to less than $500, raised by parents and teachers to print up full color postcards or brochures. Typically, principals rely on staff members with a creative bent to draw up whatever they can.

Student recruitment has always been necessary for charter schools, which are privately run but receive public money based on their enrollment, supplemented by whatever private donations they can corral.

The Harlem Success Academy network, run by the former City Council member Eva Moskowitz, is widely regarded, with admiration by some and scorn by others, as having the biggest marketing effort. Their bright orange advertisements pepper the bus stops in the neighborhood, and prospective parents receive full color mailings almost monthly.

Ms. Moskowitz said the extensive outreach was necessary to make sure they were drawing from a broad spectrum of parents. Ms. Moskowitz said they spent roughly $90 per applicant for recruitment. With about 3,600 applicants last year for the four schools in the network, she said, the total amounted to $325,000.

Charter schools, social norming and zero-sum games

You've probably heard about the Harlem Children's Zone, an impressive, even inspiring initiative to improve the lives of poor inner-city children through charter schools and community programs. Having taught in Watts and the Mississippi Delta in my pre-statistician days, this is an area of long-standing interest to me and I like a lot of the things I'm hearing about HCZ. What I don't like nearly as much is the reaction I'm seeing to the research study by Will Dobbie and Roland G. Fryer, Jr. of Harvard. Here's Alex Tabarrok at Marginal Revolution with a representative sample, "I don't know why anyone interested in the welfare of children would want to discourage this kind of experimentation."

Maybe I can provide some reasons.

First off, this is an observational study, not a randomized experiment. I think we may be reaching the limits of what analysis of observational data can do in the education debate and, given the importance and complexity of the questions, I don't understand why we aren't employing randomized trials to answer some of these questions once and for all.

More significantly I'm also troubled by the aliasing of data on the Promise Academies and by the fact that the authors draw a conclusion ("HCZ is enormously successful at boosting achievement in math and ELA in elementary school and math in middle school. The impact of being offered admission into the HCZ middle school on ELA achievement is positive, but less dramatic. High-quality schools or community investments coupled with high-quality schools drive these results, but community investments alone cannot.") that the data can't support.

In statistics, aliasing means combining treatments in such a way that you can't tell which treatment or combination of treatments caused the effect you observed. In this case the first treatment is the educational environment of the Promise Academies. The second is something called social norming.

When you isolate a group of students, they will quickly arrive at a consensus of what constitutes normal behavior. It is a complex and somewhat unpredictable process driven by personalities and random connections and any number of outside factors. You can however, exercise a great deal of control over the outcome by restricting the make-up of the group.

If we restricted students via an application process, how would we expect that group to differ from the general population and how would that affect the norms the group would settle on? For starters, all the parents would have taken a direct interest in their children's schooling.

Compared to the general population, the applicants will be much more likely to see working hard, making good grades, not getting into trouble as normal behaviors. The applicants (particularly older applicants) would be more likely to be interested in school and to see academic and professional success as a reasonable possibility because they would have made an active choice to move to a new and more demanding school. Having the older students committed to the program is particularly important because older children play a disproportionate role in the setting of social norms.

Dobbie and Fryer address the question of self-selection, "[R]esults from any lottery sample may lack external validity. The counterfactual we identify is for students who are already interested in charter schools. The effect of being offered admission to HCZ for these students may be different than for other types of students." In other words, they can't conclude from the data how well students would do at the Promise Academies if, for instance, their parents weren't engaged and supportive (a group effective eliminated by the application process).

But there's another question, one with tremendous policy implications, that the paper does not address: how well would the students who were accepted to HCZ have done if they were given the same amount of instruction * as they would have received from HCZ using public school teachers while being isolated from the general population? (There was a control group of lottery losers but there is no evidence that they were kept together as a group.)

Why is this question so important? Because we are thinking about spending an enormous amount of time, effort and money on a major overhaul of the education system when we don't have the data to tell us if what we'll spend will wasted or, worse yet, if we are to some extent playing a zero sum game.

Social norming can work both ways. If you remove all of the students whose parents are willing and able to go through the application process, the norms of acceptable behavior for those left behind will move in an ugly direction and the kids who started out with the greatest disadvantages would be left to bear the burden.

But we can answer these questions and make decisions based on solid, statistically sound data. Educational reform is not like climate change where observational data is our only reasonable option. Randomized trials are an option in most cases; they are not that difficult or expensive.

Until we get good data, how can we expect to make good decisions?

* Correction: There should have been a link here to this post by Andrew Gelman.