"No, I said 'self-select for attitude.'"

One of the many weird aspects of the charter school debate has been the non-exchanges over selection effects. Critics suggest that students who choose to go through the application process will tend to be more motivated, more committed to school, more interested in college, and more open to the idea of a heavier work load and will tend to have more stable, supportive families.

Here's an example:

Some observers, such as the authors of The Charter School Dust-Up, say that KIPP's admission process self-screens for students who are both motivated and compliant, from similarly motivated and compliant—and supportive—families. Parents must commit to a required level of involvement, which rules out badly dysfunctional families.

This would seem to be fairly straightforward. Almost every school could do a good job if it were populated solely by students who worked hard, followed instructions and had the full support of their families. A strong selection effect here could explain away most or all of the positive results observed in KIPP schools. This hypothesis even raises the possibility that KIPP schools are actually doing worse than public schools would under comparable circumstances.

(You'll notice that no mention is made of race, poverty or test scores.)

Now here's how SRI responds:

KIPP schools’ higher-than-expected test score results draw both attention and claims that they “cherry-pick” high-achieving students from poor neighborhoods. This is the first report to closely scrutinize the praise and criticisms associated with KIPP, as well as key challenges facing Bay Area KIPP schools today.

In the three KIPP schools where they were able to draw comparisons, SRI researchers found that students with lower prior achievement on the CST were more likely to choose KIPP than higher-performing students from the same neighborhood, suggesting that, at least at these schools, cherry-picking does not occur.

And here's the much-touted Mathematica report:

Our nonexperimental methods account for the pre-KIPP (baseline) characteristics of students who subsequently enter KIPP schools: not only demographic characteristics such as race/ethnicity, gender, poverty status, and special education status, but also their achievement test results for two years prior to entering KIPP. We examine the achievement levels of the students for up to four years after entering KIPP schools. In our preferred models, we compare these trajectories to the achievement trajectories of a matched set of local public school students who have similar achievement test results and demographic characteristics in the baseline period (typically third and fourth grades) but who do not enroll in KIPP.

Waiting for Superman... to get bored and drop you like a bad habit

From Roger Ebert's review of Waiting for Superman:

We have met five of these students, heard from them and their parents, and hope they'll win. The cameras hold on their faces as numbers are drawn or names are called. The odds against them are 20 to 1. Lucky students leap in joy. The other 19 of the 20 will return to their neighborhood schools, which more or less guarantees they will be part of a 50 percent dropout rate. The key thing to keep in mind is that underprivileged, inner-city kids at magnet schools such as Kipp L.A. Prep or the Harlem Success Academy will do better academically than well-off suburban kids with fancy high school campuses, athletic programs, swimming pools, closed-circuit TV and lush landscaping. [emphasis added]

Probably the most comprehensive study of the KIPP program was done by SRI, which looked at KIPP schools in the Bay Area. Here's one of their findings (again, emphasis added):

As researchers analyzed the student achievement data and KIPP’s approach, they also identified challenges facing Bay Area KIPP schools, including high student attrition rates, teacher turnover, and low state and local funding. For example, 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade.

More that any other player in the charter school movement, KIPP has relied on the cheese effect to generate its results. It's a profitable and highly praised business model built entirely of the backs of discarded students.

Was I too tough on Seyward Darby?

We've recently had a huge uptick in viewers. This is great (and I'm tremendously appreciative), but I do have to remind myself most of the people in the room walked in on the middle of the conversation.

If you do a keyword search, you would see we've been hammering away at Ms. Darby for a long time and you might get the impression that we are trying to depict her as the mad relative chained up in the New Republic's attic. We're not and she isn't. (TNR's attic position was filled long before she got there.)

Ms. Darby, a talented and well-qualified journalist, takes the brunt of the criticism because she writes the bulk of TNR's articles on education, not because those articles deviate from the magazine's position on, well, anything.

The indispensable Jonathan Cohn has handed his blog over to Darby's education posts while Jonathan Chait, arguably the best political blogger out there, has taken an almost identical position on the subject.

I addressed Chait's uncharacteristic education writing a couple of weeks ago (seems like longer) in "Strange Bedfellows":

There's too much here to cover in one post (I could do a page just on Chait's weird reaction to Samuelson's looks, a topic that I had never given any thought to up until now). I may take another pass at another section later but for now I'm going to limit myself to this particularly egregious bit:

How does Samuelson explain the existence of new charter schools that produce dramatically higher results among these lazy, no-good teenagers? He insists, "no one has yet discovered transformative changes in curriculum or pedagogy, especially for inner-city schools, that are (in business lingo) 'scalable.'" This is utterly false. The most prominent example is the Kipp schools, which have shown revolutionary improvements among poor, inner-city students and have rapidly expanded.

It is strange to see Chait take the pro-privatization side of the debate, stranger still to see him accuse critics of charter schools of having an anti-government bias*, but what pushes this into Rod Serling territory is the spectacle of having Chait, one of the most gifted bullshit detectors of the Twenty-first Century, rolling out the same sort of flawed argument that he has made a career out of dismantling.

In order to be viable, a reform has to improve on the existing system by a large enough margin to justify its implementation costs, but if you accept the metrics used by the reform movement, then you will have to conclude that charter schools do worse than public schools more often than they do better.**

So we have a major push to privatize government services which, after about two decades of testing have been shown to under-perform their traditional government-run alternatives. Rather than show why this statistic is misleading, Chait pulls out vague, anecdotal evidence of a single out-lier. Now, given the variability of the data, we would expect the top schools (or even chains) to do pretty well. That alone rebuts Chait's point, but it gets worse. Self-selection, peer effects and selective attrition*** all artificially inflate KIPP's results. When you take these factors into account, it's hard to make a compelling statistical case that even the best charter schools are outperforming public schools (though the second footnote still applies).

At the risk of over-emphasizing, this is Jonathan -- freaking -- Chait we're talking about, a writer known for his truly exceptional gift for constructing logical arguments and, more importantly, spotting the fallacies in the arguments of others. Under normal conditions, Chait would never fall for a badly presented argument-by-anomally, let alone make one, just as, under normal circumstances, a confrontation between Samuelson and Chait would result in little pieces of the former being scraped off of the walls of the Washington Post.

But Chait loses this confrontation decisively. From his ad hominem opening to his factually challenged close he fails to score a single point. And this is far from the only example of this odd reform-specific impairment affecting otherwise accomplished writers. OE has spilled endless pixels on the reform-related lapses, both statistical and rhetorical, of smart, serious, dedicated people like Chait, Seyward Darby and, of course, Ray Fisman (just do a keyword search). None of these people would normally produce the kind of work we've cataloged here. None of them would normally ignore the defection of one of the founding members of the reform movement. None of these people would normally feel comfortable dismissing without comment contradictory findings from EPI, Donald Rubin and the Rand Institute.

David Warsh has aptly made the following comparison:

Remember the recipe for a policy disaster? Start with a handful of policy intellectuals confronting a stubborn problem, in love with a Big Idea. Fold in a bunch of ambitious Ivy League kids who don’t speak the local language. Churn up enthusiasm for the program in the gullible national press – and get ready for a decade of really bad news. Take a look at David Halberstam’s Vietnam classic The Best and the Brightest, if you need to refresh your memory. Or just think back on the run-up to the war in Iraq.

but along with Halberstam, it might be time to brush off our copies of Cialdini's Influence.

From a data standpoint, the past few years have been rough on the reform movement. Charter schools have been shown to be more likely to under-perform than to outperform. Joel Klein's spectacular record turned out to be the product of creative accounting (New York City schools have actually done much worse than the rest of the state). Findings contradicting the fundamental tenets of the movement accumulated. Major figures in research (Rubin) and education (Ravitch) have publicly questioned the viability of proposed reforms.

As Cialdini lays out in great detail, when you challenge people's deeply held beliefs with convincing evidence, you usually get one of two responses. Sometimes you will actually manage to win them over. More often, though, they will dig in, embrace their beliefs more firmly and find new ways to justify them.

I think it's safe to say we don't have response number one.




* Almost all of the major tenets of the modern reform can be traced back to the Reagan era and were closely associated with the initiatives described in Franks' The Wrecking Crew.

** Ironically, if you consider the intellectual framework of the reform movement to be flawed and overly simplistic, you can actually make a much better case for charter schools.

*** From Wikipedia: "In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."

[You can find more on KIPP here]

Encouraging/Discouraging words from President Obama

(Time for another trip into N-Space -- test/retention axis)

From the President's NBC interview:

What is your message to the leadership of unions and to teacher union members?

"We want to work with you; we're not interested in imposing changes on you...you can't defend the status quo in which a third of our kids are dropping out...when you've got 2,000 schools across the country that are drop-out factories, in those schools you have to have radical change. ... The vast majority of teachers want to do a good job, they're not in it for the money. ... Ultimately if some teachers are not doing a good job they've got to go."

This is encouraging because, unlike the test-scores scare, the drop-out crisis hasn't received nearly enough attention. After gangs and school violence, it is the problem that worries me the most.

This discouraging because many of the schools being held up as models by the reformers have built much of their reputation by systematically excluding and/or chasing away the very students who drop out of traditional schools.

More on the SRI KIPP report

This is a fairly small case of researchers being (deliberately?) obtuse, but it's representative enough to merit a brief post.

First look at this passage from Wikipedia:

Some observers, such as the authors of The Charter School Dust-Up,^[5] say that KIPP's admission process self-screens for students who are both motivated and compliant, from similarly motivated and compliant—and supportive—families. Parents must commit to a required level of involvement, which rules out badly dysfunctional families.

You'll notice this is consistent with some points Joseph and I have been arguing for awhile about how selection and peer effects can affect a school.

Now pay close attention to the following from SRI:

KIPP schools’ higher-than-expected test score results draw both attention and claims that they “cherry-pick” high-achieving students from poor neighborhoods. This is the first report to closely scrutinize the praise and criticisms associated with KIPP, as well as key challenges facing Bay Area KIPP schools today.

In the three KIPP schools where they were able to draw comparisons, SRI researchers found that students with lower prior achievement on the CST were more likely to choose KIPP than higher-performing students from the same neighborhood, suggesting that, at least at these schools, cherry-picking does not occur.

Did you catch the shift? Like most areas, academic success is largely determined by attitude and work habit. This is particularly true in a school with extended hours of instruction and a longer year. There is every reason to believe that these traits are strongly selected for here, just as suggested by the Economic Policy Institute.

The Charter School Dust-Up came out from the EPI in 2005, three years before the SRI report, but the SRI researchers never directly acknowledge the EPI's conflicting findings. Instead they pull the researchers' version of the old school kid's trick of pretending to mishear an inconvenient question.

(Acknowledging) Failure Is Not an Option

Here at OE, we have been spending a lot of time discussing the horrible student attrition rates in Bay area KIPP schools, but when I first came across the statistic, I hesitated before using it in a post. The number was so absolutely damning that I had to wonder about the source. Had some anti-charter school group gone through all the data and cherry-picked the worst thing they could find?

So I looked up the source of the statistic and found a veritable sales brochure for KIPP. The article opened with the headline:

"New Study Finds San Francisco Bay Area KIPP Students Outperform Peers"

followed by the subtitle:

"Combination of Key Features Contributes to Success, Provides Lessons for Other Public Schools"

The rest of the article continues in the same vein before finally getting to this:

As researchers analyzed the student achievement data and KIPP’s approach, they also identified challenges facing Bay Area KIPP schools, including high student attrition rates, teacher turnover, and low state and local funding. For example, 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade. Annual teacher turnover rates have ranged from 18 to 49 percent since 2003-04.

Putting aside the human cost paid by the students who fell into that sixty percent and their families (which is huge), the selection/attrition process described in this post results in a school filled with roughly the top quartile of hard-working, dedicated students. All but the most incompetent administrators will have spectacular results under those circumstances. The performance described here ("In most grades, Bay Area KIPP students make above-average progress compared with national norms, and four out of five KIPP schools outperform their host district.") is, if anything, on the low end of what we would expect yet it is written up in the most glowing terms imaginable.

When it comes to charter schools, there's a long tradition of bad news stories with positive or at least neutral headlines. Consider this article from the New York Times. The key paragraph was:

But for all their support and cultural cachet, the majority of the 5,000 or so charter schools nationwide appear to be no better, and in many cases worse, than local public schools when measured by achievement on standardized tests, according to experts citing years of research. Last year one of the most comprehensive studies, by researchers from Stanford University, found that fewer than one-fifth of charter schools nationally offered a better education than comparable local schools, almost half offered an equivalent education and more than a third, 37 percent, were “significantly worse.”

And what was the headline for this grim finding?

"Despite Push, Success at Charter Schools Is Mixed"

I genuinely believe that there are some wonderful charter schools out there and we can greatly improve education by building on these successes, but in order to do that we need to recognize and honestly admit the failures.

Up until now, that honesty has been in short supply.

If reformers are looking for an indictment of our educational system...

I'd like to put some finer shadings on Joseph's recent post. I think I speak for most former and current teachers when I say that my favorite part of teaching was teaching -- explaining concepts, helping students working through problems, convincing them that they can not only do math, they can actually be good at it.

Helping a student who has shown an interest in learning always takes a high priority, but at some point something has to give. When I was teaching I routinely worked fifty plus hours a week when you count grading and making lesson plans. Add being on call ten or more hours a week and you will eventually have to make a choice. You can blow off the kids, you can blow off the grading and lesson plans or you can find out how long you can go without seeing your family or getting over six hours sleep.

As far as I can tell, the underlying assumption of the KIPP business model is an unlimited supply of Erin Gruwells. In case you've forgotten, Ms. Gruwell was the... let's say, inspiration for the character played by Hillary Swank in the movie Freedom Writers. Ms. Gruwell has managed an exceptional career but as a teacher, she followed a very common trajectory. She came into the field with high expectations and tremendous energy, worked incredibly long hours at a fantastic pace, achieved some early successes then bailed in less than five years.

KIPP burns through these teachers at an extraordinary rate ("Annual teacher turnover rates have ranged from 18 to 49 percent since 2003-04"). This is possible in large part because there are less than twenty-seven thousand students in the KIPP system. By charter school standards, this is substantial but there are over sixty-million K through 12 children in the U.S.

A number of people think KIPP is a scalable model for the nation. Many of these people hold advanced degrees from some of America's most prestigious universities. If reformers are looking for an indictment of our educational system, I can't think of a more damning one.

Make that most children left behind

In an email, Joseph accused me of burying the lede when I discussed KIPP charter schools in an earlier post. The paragraph in question was this quote from Wikipedia:

"In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."

Administrators have long known that the simplest and most reliable way to improve a school's performance is by selection and attrition of the student body. This works in the obvious, direct ways -- if you drop the kids who disrupt class and/or can't master the material, test scores and classroom management metrics will go up -- but there are at least a couple of indirect effects that are as, or more, powerful.

One is the eighty/twenty rule. Some students take more time than others and, not surprisingly, the students who lower a school's test average and management metrics are the ones who consume the most time and resources.

Even more significant are peer effects. K through 12 students are particularly sensitive to perceived social norms. By selectively removing certain students from the population, you can easily create a high degree of conformity to an almost ideal set of behaviors and attitudes.

Most educators look at getting rid of students as a last resort. The prevailing attitude is that you are there to help all the kids, not just the easy ones. You will find exceptions, of course, like principals who are a little too eager to expel certain students or find ways to influence which students are assigned to other schools, but they are forced to work around rules that discourage this behavior.

For charter schools, though, these selective factors are built into the system. Here's the deal that college prep charter schools offer, if the students go through an involved application/induction process, take more difficult courses during a longer school day and do more homework, they will have a better chance at academic and professional success. These schools have automatically limited their pool of applicants to kids who consider academic and professional success both desirable and attainable and who come from supportive, involved families that are willing to make a real effort to give their children a chance to succeed.

(quick clarification: we are talking about selection processes that favor certain attitudes and behaviors. We are NOT talking about favoring students with high test scores. Many reporters and, God help us, researchers have failed to grasp this distinction.)

Charter schools segregate out a group of students who tend to be, to put it bluntly, easy to teach. This is not an entirely bad thing. Though there are concerns about this being a zero-sum-game, there is something to be said for making sure that every neighborhood has at least one educational bright spot.

It is, however, an entirely bad thing when dishonest or naive observers evaluate these schools without taking these systemic advantages into account and it is worse still when unscrupulous administrators try to build on those systemic advantages by cooking the data through selective attrition.

If you start with a student body made up almost entirely of kids who want to be in school, avoid gangs and graduate from college, who are supported by families with the same goals, you should expect relatively low attrition. When you see the opposite, you should certainly be suspicious. Even the most inept administrator can look good if you allow him or her to pick the most promising students out of an already select group.

The 'I' stands for 'Ironic'

Check out bullet two. (From the KIPP website)

At KIPP schools:

• Parents are encouraged to be involved and to contribute;
• Teachers have the freedom to innovate and never give up on a child;
• Students work hard and come to school ready to learn.

Update: While you're here, why not take a look at the next post

Strange Bedfellows

(where your intrepid blogger sides with Robert Samuelson, gives Ray Fisman a break and wonders what in the hell happened to Jonathan Chait)

You really have to read this Jonathan Chait column on Robert Samuelson, and I don't mean that in a good way. You have to read this to see for yourself how low the educational reform movement can drag even a writer as gifted as Chait.

There's too much here to cover in one post (I could do a page just on Chait's weird reaction to Samuelson's looks, a topic that I had never given any thought to up until now). I may take another pass at another section later but for now I'm going to limit myself to this particularly egregious bit:

How does Samuelson explain the existence of new charter schools that produce dramatically higher results among these lazy, no-good teenagers? He insists, "no one has yet discovered transformative changes in curriculum or pedagogy, especially for inner-city schools, that are (in business lingo) 'scalable.'" This is utterly false. The most prominent example is the Kipp schools, which have shown revolutionary improvements among poor, inner-city students and have rapidly expanded.

It is strange to see Chait take the pro-privatization side of the debate, stranger still to see him accuse critics of charter schools of having an anti-government bias*, but what pushes this into Rod Serling territory is the spectacle of having Chait, one of the most gifted bullshit detectors of the Twenty-first Century, rolling out the same sort of flawed argument that he has made a career out of dismantling.

In order to be viable, a reform has to improve on the existing system by a large enough margin to justify its implementation costs, but if you accept the metrics used by the reform movement, then you will have to conclude that charter schools do worse than public schools more often than they do better.**

So we have a major push to privatize government services which, after about two decades of testing have been shown to under-perform their traditional government-run alternatives. Rather than show why this statistic is misleading, Chait pulls out vague, anecdotal evidence of a single out-lier. Now, given the variability of the data, we would expect the top schools (or even chains) to do pretty well. That alone rebuts Chait's point, but it gets worse. Self-selection, peer effects and selective attrition*** all artificially inflate KIPP's results. When you take these factors into account, it's hard to make a compelling statistical case that even the best charter schools are outperforming public schools (though the second footnote still applies).

At the risk of over-emphasizing, this is Jonathan -- freaking -- Chait we're talking about, a writer known for his truly exceptional gift for constructing logical arguments and, more importantly, spotting the fallacies in the arguments of others. Under normal conditions, Chait would never fall for a badly presented argument-by-anomally, let alone make one, just as, under normal circumstances, a confrontation between Samuelson and Chait would result in little pieces of the former being scraped off of the walls of the Washington Post.

But Chait loses this confrontation decisively. From his ad hominem opening to his factually challenged close he fails to score a single point. And this is far from the only example of this odd reform-specific impairment affecting otherwise accomplished writers. OE has spilled endless pixels on the reform-related lapses, both statistical and rhetorical, of smart, serious, dedicated people like Chait, Seyward Darby and, of course, Ray Fisman (just do a keyword search). None of these people would normally produce the kind of work we've cataloged here. None of them would normally ignore the defection of one of the founding members of the reform movement. None of these people would normally feel comfortable dismissing without comment contradictory findings from EPI, Donald Rubin and the Rand Institute.

David Warsh has aptly made the following comparison:

Remember the recipe for a policy disaster? Start with a handful of policy intellectuals confronting a stubborn problem, in love with a Big Idea. Fold in a bunch of ambitious Ivy League kids who don’t speak the local language. Churn up enthusiasm for the program in the gullible national press – and get ready for a decade of really bad news. Take a look at David Halberstam’s Vietnam classic The Best and the Brightest, if you need to refresh your memory. Or just think back on the run-up to the war in Iraq.

but along with Halberstam, it might be time to brush off our copies of Cialdini's Influence.

From a data standpoint, the past few years have been rough on the reform movement. Charter schools have been shown to be more likely to under-perform than to outperform. Joel Klein's spectacular record turned out to be the product of creative accounting (New York City schools have actually done much worse than the rest of the state). Findings contradicting the fundamental tenets of the movement accumulated. Major figures in research (Rubin) and education (Ravitch) have publicly questioned the viability of proposed reforms.

As Cialdini lays out in great detail, when you challenge people's deeply held beliefs with convincing evidence, you usually get one of two responses. Sometimes you will actually manage to win them over. More often, though, they will dig in, embrace their beliefs more firmly and find new ways to justify them.

I think it's safe to say we don't have response number one.




* Almost all of the major tenets of the modern reform can be traced back to the Reagan era and were closely associated with the initiatives described in Franks' The Wrecking Crew.

** Ironically, if you consider the intellectual framework of the reform movement to be flawed and overly simplistic, you can actually make a much better case for charter schools.

*** From Wikipedia: "In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."