Substituting metrics for judgement

This episode of This American Life concludes with a fascinating account of the manipulation of crime statistics. Drop by, take a listen, give them a buck.

If reformers are looking for an indictment of our educational system...

I'd like to put some finer shadings on Joseph's recent post. I think I speak for most former and current teachers when I say that my favorite part of teaching was teaching -- explaining concepts, helping students working through problems, convincing them that they can not only do math, they can actually be good at it.

Helping a student who has shown an interest in learning always takes a high priority, but at some point something has to give. When I was teaching I routinely worked fifty plus hours a week when you count grading and making lesson plans. Add being on call ten or more hours a week and you will eventually have to make a choice. You can blow off the kids, you can blow off the grading and lesson plans or you can find out how long you can go without seeing your family or getting over six hours sleep.

As far as I can tell, the underlying assumption of the KIPP business model is an unlimited supply of Erin Gruwells. In case you've forgotten, Ms. Gruwell was the... let's say, inspiration for the character played by Hillary Swank in the movie Freedom Writers. Ms. Gruwell has managed an exceptional career but as a teacher, she followed a very common trajectory. She came into the field with high expectations and tremendous energy, worked incredibly long hours at a fantastic pace, achieved some early successes then bailed in less than five years.

KIPP burns through these teachers at an extraordinary rate ("Annual teacher turnover rates have ranged from 18 to 49 percent since 2003-04"). This is possible in large part because there are less than twenty-seven thousand students in the KIPP system. By charter school standards, this is substantial but there are over sixty-million K through 12 children in the U.S.

A number of people think KIPP is a scalable model for the nation. Many of these people hold advanced degrees from some of America's most prestigious universities. If reformers are looking for an indictment of our educational system, I can't think of a more damning one.

Cost of Education

I was speaking with Mark about this post and an interesting point came up. Teachers already have long and structured hours (pre and post class supervision, class hours and lunch duty) plus work at home (marking -- which at the high school level is never finished during the workday). KIPP has teachers also available by cell phone. In the private sector it generally involves a pretty hefty pay increase to put an associate on a pager.

When we talk about the need to increase our investment in education, are we really prepared to increase either the number of teachers or the median salary of teachers? After all, we are currently talking about teacher layoffs. Perhaps Mark, who has more practical experience in this matter, can comment?

Reflections on the KIPP posts

I want to explicitly emphasize this point in Mark's last post:

One is the eighty/twenty rule. Some students take more time than others and, not surprisingly, the students who lower a school's test average and management metrics are the ones who consume the most time and resources.

In the light of the KIPP website (which he discusses here):

Having teachers available by cell phone after school for homework help

Now, let us consider two effects of this policy. One, the school day is a lot longer for a teacher than it used to be. Add in the rigorous extracurricular activities and the (seemingly 24 hour on call status) and at best you are creating an environment where investing more resources in teaching creates better outcomes.

But I don't think that this is going to be the main effect. If you consider Mark's eighty/twenty rule, you create enormous incentives for teachers to gently nudge out the student who calls for an hour and half to discuss homework every single evening. Even if the teachers at KIPP don't do this (a proposition of which I would be highly sceptical), I would be deeply concerned about what whether this approach was scalable.

It think it would be critical to determine if an approach was scalable before basing massive reforms off of it. I am happy for innovation and improvement in education to continue but I think that we should proceed carefully and based on the evidence. In education, for reasons that evade me, people seem to be very non-critical about the evidence.

Make that most children left behind

In an email, Joseph accused me of burying the lede when I discussed KIPP charter schools in an earlier post. The paragraph in question was this quote from Wikipedia:

"In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."

Administrators have long known that the simplest and most reliable way to improve a school's performance is by selection and attrition of the student body. This works in the obvious, direct ways -- if you drop the kids who disrupt class and/or can't master the material, test scores and classroom management metrics will go up -- but there are at least a couple of indirect effects that are as, or more, powerful.

One is the eighty/twenty rule. Some students take more time than others and, not surprisingly, the students who lower a school's test average and management metrics are the ones who consume the most time and resources.

Even more significant are peer effects. K through 12 students are particularly sensitive to perceived social norms. By selectively removing certain students from the population, you can easily create a high degree of conformity to an almost ideal set of behaviors and attitudes.

Most educators look at getting rid of students as a last resort. The prevailing attitude is that you are there to help all the kids, not just the easy ones. You will find exceptions, of course, like principals who are a little too eager to expel certain students or find ways to influence which students are assigned to other schools, but they are forced to work around rules that discourage this behavior.

For charter schools, though, these selective factors are built into the system. Here's the deal that college prep charter schools offer, if the students go through an involved application/induction process, take more difficult courses during a longer school day and do more homework, they will have a better chance at academic and professional success. These schools have automatically limited their pool of applicants to kids who consider academic and professional success both desirable and attainable and who come from supportive, involved families that are willing to make a real effort to give their children a chance to succeed.

(quick clarification: we are talking about selection processes that favor certain attitudes and behaviors. We are NOT talking about favoring students with high test scores. Many reporters and, God help us, researchers have failed to grasp this distinction.)

Charter schools segregate out a group of students who tend to be, to put it bluntly, easy to teach. This is not an entirely bad thing. Though there are concerns about this being a zero-sum-game, there is something to be said for making sure that every neighborhood has at least one educational bright spot.

It is, however, an entirely bad thing when dishonest or naive observers evaluate these schools without taking these systemic advantages into account and it is worse still when unscrupulous administrators try to build on those systemic advantages by cooking the data through selective attrition.

If you start with a student body made up almost entirely of kids who want to be in school, avoid gangs and graduate from college, who are supported by families with the same goals, you should expect relatively low attrition. When you see the opposite, you should certainly be suspicious. Even the most inept administrator can look good if you allow him or her to pick the most promising students out of an already select group.

The 'I' stands for 'Ironic'

Check out bullet two. (From the KIPP website)

At KIPP schools:

• Parents are encouraged to be involved and to contribute;
• Teachers have the freedom to innovate and never give up on a child;
• Students work hard and come to school ready to learn.

Update: While you're here, why not take a look at the next post

Rationality

From Econned page 148:

Greenspan was hardly alone in his dogmatic belief in the wisdom of leaving “free markets” to their own devices . . . Judge Frank Easterbrook and Daniel Fishel, then University of Chicago Law School dean, asserted in 1991: “[A] law against fraud is not an essential or necessarily important ingredient of securities markets.”

I find this line of thinking to be rather remarkable. The two classic libertarian roles for government are to prevent force and fraud. Markets can’t be efficient if information is missing or removed – in a world where identities can be re-invented then you simply can’t have a national (let alone) global market if fraud is a permissible business strategy. The only way to trust people with your money or in business deals would be a long history of relationships.

This line of thinking requires one to think that there are no information asymmetries in the real world. Furthermore, a rational agent permitted to engage in fraud might be willing to destroy their relationship for a large enough pay-off. Yves Smith gives an example of how this thinking is leads to odd conclusions:

Easterbrook went so far as blocking a plaintiff from presenting a case that argued that an auditor has assisted in a fraud. The judge claimed it would be “irrational” for the accountant to behave in that way, given his interest in preserving his reputation.

Of course, one could argue that all potential criminals have an interest in preserving their reputation. But one might very well risk their reputation for a sufficiently high pay-off – especially if one is highly unlikely to be prosecuted.

I do not think that this type of thinking is universal among “free market” advocates but it is extremely concerning that these arguments can be advanced without public mockery.

More from Yves Smith

An interesting point in Econned (pages 223-224):

One way to contain compensation is for the central bank to raise interest rates when inflation starts to build. The logic is that increasing unemployment will moderate pay pressures and also discourage business from giving employers pay increases in excess of productivity gains. Ironically, quite a few “free markets” supporters endorse this type of intervention to correct a perceived marker failure (labor having undue bargaining power) but reject a raft of others.

One of the hard things about free markets is that everybody seems to want to meddle with at least one small portion that they see as problematic. However, if you are not careful you can end up with unexpected consequences as markets respond to incentives in unexpected ways. This is not to say that inflation is a good thing (it isn’t) but rather to recognize that real markets have limitations and failures.

This example is particularly interesting as one of the great issues of the moment is how powerless workers have become in the face of tight credit markets.

Fishing expeditions

This has to be a bad idea. Chronic pain is a difficult condition to treat already with an extremely limited set of options; second guessing treatment decisions seems to be likely to lead to more suffering in the long run.

Changing statistical languages

Even when a current programming language has drawbacks, it can be hard to change to a more optimal language due to the investment in the current language. Here is a comment from Julien Cornebise:

But R is here and in everyday use, and the matter is more of making it worth using, to its full potential. I have no special attachment to R, but any breakthrough language that would not be entirely compatible with the massive library contributed over the years would be doomed to fail to pick-up the everyday statistician—and we’re talking here about far-fetched long-term moves. Sanitary breakthrough, but harder to make happen when such an anchor is here.

R is a pretty amazing language and, as a long term SAS user, I must admit that I am delighted by the graphics and the cool packages. Of course, the dark side of such a rich library is needing to learn about the reliability and limitations of all of the different packages.

H/t: Andrew Gelman

Strange Bedfellows

(where your intrepid blogger sides with Robert Samuelson, gives Ray Fisman a break and wonders what in the hell happened to Jonathan Chait)

You really have to read this Jonathan Chait column on Robert Samuelson, and I don't mean that in a good way. You have to read this to see for yourself how low the educational reform movement can drag even a writer as gifted as Chait.

There's too much here to cover in one post (I could do a page just on Chait's weird reaction to Samuelson's looks, a topic that I had never given any thought to up until now). I may take another pass at another section later but for now I'm going to limit myself to this particularly egregious bit:

How does Samuelson explain the existence of new charter schools that produce dramatically higher results among these lazy, no-good teenagers? He insists, "no one has yet discovered transformative changes in curriculum or pedagogy, especially for inner-city schools, that are (in business lingo) 'scalable.'" This is utterly false. The most prominent example is the Kipp schools, which have shown revolutionary improvements among poor, inner-city students and have rapidly expanded.

It is strange to see Chait take the pro-privatization side of the debate, stranger still to see him accuse critics of charter schools of having an anti-government bias*, but what pushes this into Rod Serling territory is the spectacle of having Chait, one of the most gifted bullshit detectors of the Twenty-first Century, rolling out the same sort of flawed argument that he has made a career out of dismantling.

In order to be viable, a reform has to improve on the existing system by a large enough margin to justify its implementation costs, but if you accept the metrics used by the reform movement, then you will have to conclude that charter schools do worse than public schools more often than they do better.**

So we have a major push to privatize government services which, after about two decades of testing have been shown to under-perform their traditional government-run alternatives. Rather than show why this statistic is misleading, Chait pulls out vague, anecdotal evidence of a single out-lier. Now, given the variability of the data, we would expect the top schools (or even chains) to do pretty well. That alone rebuts Chait's point, but it gets worse. Self-selection, peer effects and selective attrition*** all artificially inflate KIPP's results. When you take these factors into account, it's hard to make a compelling statistical case that even the best charter schools are outperforming public schools (though the second footnote still applies).

At the risk of over-emphasizing, this is Jonathan -- freaking -- Chait we're talking about, a writer known for his truly exceptional gift for constructing logical arguments and, more importantly, spotting the fallacies in the arguments of others. Under normal conditions, Chait would never fall for a badly presented argument-by-anomally, let alone make one, just as, under normal circumstances, a confrontation between Samuelson and Chait would result in little pieces of the former being scraped off of the walls of the Washington Post.

But Chait loses this confrontation decisively. From his ad hominem opening to his factually challenged close he fails to score a single point. And this is far from the only example of this odd reform-specific impairment affecting otherwise accomplished writers. OE has spilled endless pixels on the reform-related lapses, both statistical and rhetorical, of smart, serious, dedicated people like Chait, Seyward Darby and, of course, Ray Fisman (just do a keyword search). None of these people would normally produce the kind of work we've cataloged here. None of them would normally ignore the defection of one of the founding members of the reform movement. None of these people would normally feel comfortable dismissing without comment contradictory findings from EPI, Donald Rubin and the Rand Institute.

David Warsh has aptly made the following comparison:

Remember the recipe for a policy disaster? Start with a handful of policy intellectuals confronting a stubborn problem, in love with a Big Idea. Fold in a bunch of ambitious Ivy League kids who don’t speak the local language. Churn up enthusiasm for the program in the gullible national press – and get ready for a decade of really bad news. Take a look at David Halberstam’s Vietnam classic The Best and the Brightest, if you need to refresh your memory. Or just think back on the run-up to the war in Iraq.

but along with Halberstam, it might be time to brush off our copies of Cialdini's Influence.

From a data standpoint, the past few years have been rough on the reform movement. Charter schools have been shown to be more likely to under-perform than to outperform. Joel Klein's spectacular record turned out to be the product of creative accounting (New York City schools have actually done much worse than the rest of the state). Findings contradicting the fundamental tenets of the movement accumulated. Major figures in research (Rubin) and education (Ravitch) have publicly questioned the viability of proposed reforms.

As Cialdini lays out in great detail, when you challenge people's deeply held beliefs with convincing evidence, you usually get one of two responses. Sometimes you will actually manage to win them over. More often, though, they will dig in, embrace their beliefs more firmly and find new ways to justify them.

I think it's safe to say we don't have response number one.




* Almost all of the major tenets of the modern reform can be traced back to the Reagan era and were closely associated with the initiatives described in Franks' The Wrecking Crew.

** Ironically, if you consider the intellectual framework of the reform movement to be flawed and overly simplistic, you can actually make a much better case for charter schools.

*** From Wikipedia: "In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."

Non-compete agreeements

Felix Salmon is discussing the lawsuit over Mark Hurd accepting a job at Oracle. The crux of the argument seems to be concerns that Mr. Hurd might reveal confidential information as part of his new job. Fortunately, for him, the state of California takes a very dim view of non-compete agreements. From commenter Vania, in the comments to Mr. Salmon's post:

The covenant not to compete, as written, is simply unenforceable under California Business & Professions Code Section 16600:

“16600. Except as provided in this chapter, every contract by which anyone is restrained from engaging in a lawful profession, trade, or business of any kind is to that extent void.”

Now I have a generally dim view of non-compete agreements, so I think that this law is good sense. While I don't have any issue with a CEO having more restrictive rules, given the level of compensation that they are given, I am dubious that these agreements are given only at the CEO level. I know that I was under one, once, in a very junior position at a firm.

The key issue is that I am unsure of how easily one can freely consent to such an agreement in the midst of a dismissal. The person has just had their life turned upside down and likely lost a crucial income stream. The company has had time to prepare the exit package and carefully optimize it for their interests. They have had lawyers look it over and had HR vet the relevant policies. I am unclear that these structural differences in information can be overcome nor do I really see the "sign this or get no severance" as being a real choice for people who have had no ability to assess their options.

At an even more fundamental level, I am unclear why these sorts of agreements don't violate our norms of a free market. How can there be a bigger barrier to free economic activity than a pledge for workers not to sell their skills? We are already worried about issued with implied compensation, does this not count as a hidden cost that is not declared up front?

At best one might argue for contract law and the ability of people/organizations to enter into agreements. But the santicity of contract law seems to be under attack when it favors the worker (consider tenure and defined benefit pension plans). Why do the same arguments about net social good not apply here?

Vintage of Educational Reform

I just wanted to make a point in the education debate in passing. It is well worth re-reading Mark's post on his experience with these same reforms:

The trouble is that almost none of the people using the term 'reform' are actually suggesting any reforms. Most of the proposals that have been put forward are simply continuations or extensions of the same failed policies and questionable theories that have been coming out of schools of education for years, if not decades.

In 1994, RAND did a policy brief on the use of standardized scores in education. Look at the conclusions (from 16 years ago):

Research has not been able to pinpoint the effects of the noneducational influences. Nevertheless, people have misused test-score data in the debate to give education a "bad rap." Koretz lists three broad, overlapping kinds of misuse that should be avoided in honest, future debate:

1.Simplistic interpretations of performance trends: These trends should not be taken at face value, ignoring the various factors that influence them: for example, demographic changes in test takers or inflation of scores caused by test-based accountability.

2.Unsupported "evaluations" of schooling: Simple aggregate scores are not a sufficient basis for evaluating education--unless they provide enough information to rule out noneducational influences on performance. Most test-score databases do not offer that kind of information.

3.A reductionist view of education: Koretz notes that it may be "trite" but it is true that education is a "complex mix of successes and failures . . . what works in one context or for one group of students may fail for another." Unfortunately, that truism is often ignored. For example, in the early 1980s, when people were reasonably concerned about falling aggregate test scores, they asked for wholesale changes in policies, without first asking which policies most needed changing or which students or schools most needed new policies.

Maybe we should think twice before declaring that drops or stagnation in scores are necessarily evidence that education is failing? As for the performance gains in charter schools, Mark has been after that issue from the beginning of this discussion here at OE.

It's worth keeping this in mind when evaluating calls for massive reforms on the basis of standardized test scores alone.

Interesting thought

From Grognardia:

Indeed, if there's one "problem" with reading a book like Galactic Patrol nowadays it's that, as the wellspring of so much that came after, it appears trite and unoriginal when, in point of historical fact, everything else that followed it is what's trite and unoriginal. The "Lensmen" series is big and bold and, while I'd never argue that it's scientific speculations hold much water (though, to be fair, many of its ideas were based on the science of its day), it's nevertheless a fun read. There can be no doubt why it exerted such a profound influence on the imaginations of later authors in the genre.

I think that this is a very insightful point that brings up a good point when reading key historical works. Good ideas spawn a lot of imitators and some of these will be better (in terms of some aspects) than the original works. Just think of how many of the imitators of Lord of the Rings are better at pacing the plot! But who among them has such a rich and unique vision of an alternate world?

But these are the works that change entire genres of fiction.

Education in Canada

Another interesting point from Worthwhile Canadian Initiative:

The demand for French immersion education in Vancouver so far outstrips the supply that the school board allocates places by lottery.

But why? Is it because French is a useful employment skill? Because learning to speak French makes you a better person? Or is it because parents know intuitively what economists can show econometrically: peer effects matter. Being with high achieving peers raises a student's own achievement level.

Consider this point quoted in that article:

If students with special needs were equally distributed among all classes, each teacher would on average have 3.4 students with special needs. However, in schools that have early immersion programs, the average in core English programs is about 5.7 students.

I think that it is quite possible that these results could be applied to charter schools with long waiting lists in the United States. Why does this matter?

Because the use of lotteries could otherwise be considered to be a form of randomization. But it seems odd that being taught in a second language would result in better educational outcomes, per se. Which suggests that Frances Woolley has a point that peer effects really do matter and we should consider this when evaluating outcomes in US charter schools.

Update: Of course, I forget to mention that Mark brought up this exact point at the beginning of our foray into education.