Alice in Lawyerland: would the laws Disney lobbied for have prevented Disney from existing in the first place?

(disclaimer: I have cashed a number of royalties checks over the years so the following is obviously not an attack on the concept of intellectual property. I like royalty checks. I'm just worried about the consequences of taking these things to an extreme.)

In 1998, the Walt Disney company had a problem: their company mascot was turning 70. Mickey Mouse had debuted in 1928's "Mickey Mouse In Plane Crazy" which meant that unless something was done, Mickey would enter the public domain within a decade. This was a job for lobbyists, lots of lobbyists.

From Wikipedia:

The Copyright Term Extension Act (CTEA) of 1998 extended copyright terms in the United States by 20 years. Since the Copyright Act of 1976, copyright would last for the life of the author plus 50 years, or 75 years for a work of corporate authorship. The Act extended these terms to life of the author plus 70 years and for works of corporate authorship to 120 years after creation or 95 years after publication, whichever endpoint is earlier. Copyright protection for works published prior to January 1, 1978, was increased by 20 years to a total of 95 years from their publication date.

This law, also known as the Sonny Bono Copyright Term Extension Act, Sonny Bono Act, or pejoratively as the Mickey Mouse Protection Act,[2] effectively "froze" the advancement date of the public domain in the United States for works covered by the older fixed term copyright rules. Under this Act, additional works made in 1923 or afterwards that were still copyrighted in 1998 will not enter the public domain until 2019 or afterward (depending on the date of the product) unless the owner of the copyright releases them into the public domain prior to that or if the copyright gets extended again. Unlike copyright extension legislation in the European Union, the Sonny Bono Act did not revive copyrights that had already expired. The Act did extend the terms of protection set for works that were already copyrighted, and is retroactive in that sense.

Mickey had been Disney's biggest hit but he wasn't their first. The studio had established itself with a series of comedies in the early Twenties about a live-action little girl named Alice who found herself in an animated wonderland. In case anyone missed the connection, the debut was actually called "Alice's Wonderland." The Alice Comedies were the series that allowed Disney to leave Kansas and set up his Hollywood studio.

For context, Lewis Carroll published the Alice books, Wonderland and Through the Looking Glass, in 1865 and 1871 and died in 1898. Even under the law that preceded the Mouse Protection Act, Alice would have been the property of Carroll's estate and "Alice's Wonderland" was a far more clear-cut example of infringement than were many of the cases Disney has pursued over the years.

In other words, if present laws and attitudes about intellectual property had been around in the Twenties, the company that lobbied hardest for them might never have existed.

There's nothing unusual about a small company or start-up exploiting lapsed or unenforced copyrights to get a foothold. The public domain has long been fertile ground for stage companies, record companies, publishers, and producers of movies or radio and television; it's just been getting a lot less fertile lately.

Instrumental Variables

Reading this post by Andrew Gelman got me thinking: how do we really know if a variable is an instrument? Demonstrating that a variable is an instrument often seems to be a matter of telling a compelling clinical story but leaves one with strong and unverifiable assumptions.

This is a real issue with the Physician Preference instrument in pharmacoepidemiology where it has the potential to be either a major advance or a blind alley. But how would one know for sure? I suspect that it is more likely the former than the latter but sorting this out with any level of certainty isn't easy.

Any insights out there?

Study: Teacher bonuses don't affect student tests

from the AP:

ATLANTA – A study released Tuesday found that offering performance bonuses to teachers does nothing to raise test scores, raising doubts about the viability of the Obama administration's push for merit pay to improve education.

The study released Tuesday by Vanderbilt University's National Center on Performance Incentives researchers found that students in classrooms where teachers received bonuses saw the same gains as the classes where educators got no incentive.

"I think most people agree today that the current way in which we compensate teachers is broken," said Matthew Springer, executive director of the Vanderbilt center and lead researcher on the study. "But we don't know what the better way is yet."

The ultimate superpower is litigation

As part of our ongoing series on the surreal world of intellectual property (see the last post), here's a fun fact from comic book writer and historian Don Markstein:

Marvel and DC Comics are arch-rivals when it comes to market share in the comic book industry. But they're capable of an amazing degree of cooperation when it comes to maintaining their shared position as the industry's leaders. One of the ways they cooperate is in maintaining a joint trademark on the word "superhero" — as if Charlton, Harvey, Archie, ACG and Gold Key, to name only a few of the dozens that used both the genre and the word back before they established their mutual legal hegemony over it, never existed — to say nothing of Dark Horse, Image and others that use the genre today and aren't permitted to use the word.

In fact, here's a Dell comic book that actually made the term its very title, long before Marvel and DC decided to sew up rights to it. This should serve as a beacon to anyone willing to attempt to withstand the mighty onslaught of their lawyers, and point out that the word was in general use, with nobody even attempting to prevent others' using it, even before Superman.

Graph of the Day -- Copyrights

From Wikipedia:

I'll have more on this later.

Murphy strikes again

If you have ever wondered what else could go wrong, this is a great example.

What makes it tricky is how many different sets of rights need to be balanced. Clearly tenants need time to be able to move. It's obvious that owners need to be able to sell if ownership is to have any real meaning. And the new owners need to be able to actually take possession of their property. But somehow an outcome has happened that is sub-optimal for all involved.

What is it about real estate that makes it a locus of difficult transactions?

Today's Napoleon quote

In handy comics form:

"Heroin's doing the heavy lifting"

I'm pretty sure I'll quote this some time in the near future.


From Kumail Nanjiani:



I first heard this on This America Life. If you enjoyed it, go by and give them a buck. They're good people.

What is rich?

This was a very interesting post by Brad Delong. There is a response by the original author here.

One thing strikes me immediately. If we look at the expected expenses of this particular person, the $60,000 per year in private school costs immediately jumps out as interesting. Perhaps part of why our public schools are having issues is that the Upper Middle class has pulled out of them?

Another thought is interesting; the author suggests that:

Returning to those “small government” days of 2000 would be a start, as this would result in a huge cut in the size of government.

But appears to be against returning to the same tax levels as 2000. At some point I would really like to see more concrete proposals for what to cut in order to make lower taxes a sustainable option.

Alex Tabarrok's literary leverage

There's a very good discussion about superstars and the winner-take-all economy over at Economist's View. Alex Tabarrok argues that the winner-take-all effect in literature can be explained by factors like technology and venue size:

J.K. Rowling is the first author in the history of the world to earn a billion dollars. I do not disparage Rowling when I say that talent is not the explanation for her monetary success. Homer, Shakespeare and Tolkien all earned much less. Why? Consider Homer, he told great stories but he could earn no more in a night than say 50 people might pay for an evening's entertainment. Shakespeare did a little better. The Globe theater could hold 3000 and unlike Homer, Shakespeare didn't have to be at the theater to earn. Shakespeare's words were leveraged.

Tolkien's words were leveraged further. By selling books Tolkien could sell to hundreds of thousands, even millions of buyers in a year - more than have ever seen a Shakespeare play in 400 years. And books were cheaper to produce than actors which meant that Tolkien could earn a greater share of the revenues than did Shakespeare (Shakespeare incidentally also owned shares in the Globe.)

Rowling has the leverage of the book but also the movie, the video game, and the toy. And globalization, both economic and cultural, means that Rowling's words, images, and products are translated, transmitted and transported everywhere - this is the real magic of Ha-li Bo-te.

But it's possible to look at these examples in an entirely different way.

There's no question that technology and the ability to leverage creative works has a tremendous effect on the economics (and therefore the content) of popular culture, but how well does this particular account support that conclusion?

Homer is a bad example partially because he probably never existed, but mainly because the model Prof. Tabarrok describes, traveling performers working small venues, didn't really apply to writers at all. These minstrels were simply repeating stories that they had accumulated. The closest analogy today would be a cover band working bars and small clubs. (Hesiod throws in a bit of a monkey wrench here, but that's a topic for another day.)

If you skip ahead two or three hundred years you do have successful writers like Sophocles having their works performed in large venues. Though it's difficult to draw an analogy between forms of compensation then and now, they were certainly well rewarded for their work. Go on to the Roman era you have successful writers producing book length poems and even novels despite the lack of printing presses.

As for Shakespeare sticking to the stage, this had little to do with the relative cost of books and actors. There were the equivalent of cheap paperback versions of Shakespeare's plays published during his lifetime. There were also productions of his plays away from the Globe. Shakespeare's words were widely leveraged. The problem wasn't technology or venue; it was the lack of modern copyright laws. The revenue went to other people.

Tolkien is a bad example for other reasons. His body of work is small. His books were difficult to translate into other media. Significant sales didn't start until years after the books were written (prompted, in part, by the mistaken belief that the copyright was limited to Britain).

A better example would be Erle Stanley Gardner who had a large body of work, sold more books than Tolkien and was, during his lifetime, adapted into movies, TV, radio, comic books and probably a few other media. Was Rowling better leveraged than Gardner? Sure, but not by as much as you might think.

There is obviously more behind the rise of the superstar author than technology and the ability to leverage words, more than I have time to address now, but if I were to pursue it, I think I'd make the case for this being a story of lobbying and government regulation of the market in the form of copyrights. Technology has changed, but so has the law.




And now, just in case any of the above might be read as a slight against Rowling, I'll let Stephen King have the last word with his comparison of Harry Potter and the Twilight books:

Both Rowling and Meyer, they’re speaking directly to young people… The real difference is that Jo Rowling is a terrific writer and Stephenie Meyer can’t write worth a darn. She’s not very good.

OT: More table gaming thoughts

A Paladin in Citadel has an interesting post on how experience points where awarded in the pre-dungeons and dragons proto-role playing games. Note:

In the fantasy game originally played by Arneson, it was primarily through the recovery, and appropriate expenditure, of long-lost treasure hoards, that characters advanced in levels. Appropriate expenditure is a critical component for all classes, as it is only through the expenditure of gold (and the Wizard's case, both expenditure of gold for the spell-making materials, and time, in creating his spells) in ways meaningful to the character's motivations and interests that the characters can advance.


If you have not taken note yet, let me draw something striking about this experience points system to your attention now. No experience points for monsters killed in D @ D.

I think that this approach has some advantages. One, is that it militates against the recovered treasure being used to shop for the newest and coolest magic item. To expend gold in this manner is to reduce the rate of character advancement. Two, it gives character's goals outside of their adventuring life and makes the creation of a game world a necessary part of character development. Finally, it allows the monsters to remain scary and difficult to fight (as a major goal is to avoid combat, especially if it is mostly risk with little reward).

In more recent edition of dungeons and dragons, advancement is based on either defeating opponents or advancing the plot. However, plot advancement can reduce the options that players have and make them feel less in control of the direction of the game. On the other had, defeating monsters means that the monsters have to be relatively weak in every single combat. After all, with between 10 and 30 encounters per level, a high level character needs to be the victor of hundreds of combats. Not even the great duellists of history tended to manage that!

There is another (subtle) advantage to this approach -- high-level characters will tend to be accomplished in the world. High level wizards will run laboratories and teach apprentices because that is how they generate experience. High level clerics will have supported and developed churches. Even high level fighters will have connection to the world (perhaps as petty nobility) through their investment of their adventuring loot.

It's an interesting system . . .

KIPP and the Teach(er) for America (or you don't know Jake)

Sometimes the most instructive passages are the most painful to read. This account from the New York Times had me wincing every third word (though I will admit to a little schadenfreude at seeing the top of her class at Stanford being outmaneuvered by Jake from Two and a Half Men).

Time permitting I'll do a post on why this was such a disastrous lesson and how a better, more experienced teacher would have done things differently. For now, though, let's approach it from the other side: what would happen if we kept Ms. Nguyen and lost Jake. Putting aside for a moment peer effects, assuming no one else steps up to take the role, how would the Jake-less class be different?

The lesson plan would still be weak, but this is a math class and most of the actual learning in a math class takes place after the lesson when the students start on their worksheets and homework. With more time and a less adversarial relationship with the class, the teacher can go from desk to desk, checking to make sure that problems are being done correctly and helping the students who are having trouble.

Now let's add in the peer effects. Jake has reset the norms of behavior for the class. He has also established that it is possible to jerk the teacher's chain and create great entertainment value with few negative consequences. In a Jake-less class this wouldn't be an issue. The inability to assert authority is only an issue when someone questions it.

In short, losing Jake should produce a substantial gain in student performance and classroom metrics.

Charter schools are designed to be Jake-free zones but none of the effects of removing Jake are likely to show up in the lottery-based analyses so favored by charter school supporters. This creates a fatally flawed set of metrics.

Worse yet, it creates a system of reforms that have, too often, based their claims of success on leaving behind the very students who needed the most help.

Complex Systems

Libertarian Megan McArdle is in favor of education reform. However, in her latest column she points out the major obstacle faced by reformers:

That's why radical reformers so often end up vilified (and frankly, so often end up making some colossal bloomers; institutions are complicated, and reformer's prescriptions are never as complex as the institution they are trying to change).

I think observations like this elevate the debate. They raise the legitimate question of how do we improve a complex system (my view: incrementally and experimentally). But, as Mark has noted, the debate has often drifted into the surreal with even staunch leftists desperately opposing government regulation of education. It is an odd debate.

XKCD understates its case

There's no reason to resort to a double negative. Just check this out.

At last, something that Paul Krugman and Alex Tabarrok can agree on

Though it involves Middle Earth, not macroeconomics.

One reason to look forward to the Republicans retaking Congress and the White House

At least liberals can be happy again.

A day in the life of a Teach(er) for America -- posted for future comment

I've been meaning to post this for a long time (from Michael Winerip's NYT article on Teach for America):

The 774 new recruits who are training here are housed in Rice University dorms. Many are up past midnight doing lesson plans and by 6:30 a.m. are on a bus to teach summer school to students making up failed classes. It’s a tough lesson for those who’ve come to do battle with the achievement gap.

Lilianna Nguyen, a recent Stanford graduate, dressed formally in high heels, was trying to teach a sixth-grade math class about negative numbers. She’d prepared definitions to be copied down, but the projector was broken.

She’d also created a fun math game, giving every student an index card with a number. They were supposed to silently line themselves up from lowest negative to highest positive, but one boy kept disrupting the class, blurting out, twirling his pen, complaining he wanted to play a fun game, not a math game.

“Why is there talking?” Ms. Nguyen said. “There should be no talking.”

“Do I have to play?” asked the boy.

“Do you want to pass summer school?” Ms. Nguyen answered.

The boy asked if it was O.K. to push people to get them in the right order.

“This is your third warning,” Ms. Nguyen said. “Do not speak out in my class.”

Lots of interesting jumping off points here about Teach for America, classroom life and how not to handle a math lesson. Now if I can just find time to write them up.

"We've made huge advances in what they're called" -- New Republic edition

This post by Joseph got me thinking. Charter schools are private contractors providing services that were previously provided by the government. Any statement that's true about charter schools should still be true if you substitute in the phrase "some private contractors."

But if you actually make the substitution, you often end up with statements the author would never think of making. Statements like this:

But Mead says ... she’s seen Gray hint that he’d like to more tightly regulate [private contractors]. “We have a law that gives a tremendous amount of autonomy to the [private contractors] but enables them to implement programs that can be effective. If you try to put more regulation on that, if can dissuade people from [privatizing],” Mead says.

Would Seyward Darby normally describe a push for tighter regulation of private contractors as "disappointing"? Would the New Republic normally endorse a candidate because he was against stricter regulation of private contractors? Would everyone take a moment and see if Rod Serling is taking a smoke break in the vicinity?

I strongly believe that there is a place for charter schools in our system, but those schools have to meet exactly the same criteria as other contractors. Two of those criteria are transparency and openness to regulation, and given recent history, it's safe to say that some charter schools are failing these tests.

[note: typo in the title has been corrected]

Tenure Track Jobs

Somehow the blogroll for Prof-like Substance isn't updating so I missed this post. But it is excellent advice. The worst thing that can happen is that people decline your applications. But it can be remarkable how somebody can end up as a great fit in a place that they never expected . . .

Implications of the DC primary on education

Mark and I have been talking a lot about education. Today Seyward Darby has an article on the implications of Michelle Rhee leaving. What struck me as most interesting is that there are three concerns raised:

1) The fate of the new teacher evaluation system

2) That performance pay, based on IMPACT score, might not go forward

3) There could be increased regulation of character schools

Mark has commented on issues with a focus on narrow methods of teaching evaluation. But I think it is safe to say that items 1 and 2 depend critically on the quality of the method used.

As for number 3, deregulated private educational entities have geenrated concerns as the quality of instruction is hard to measure.

So everything hinges on how good the IMPACT evaluation score is. Now, this score is a hybrid (as opposed to pure standardized test scores) so it might not be awful:

It rates its teachers based partly on how well they improve student learning from year to year, and partly on intensive classroom observation by their supervisors.

However, it does not address the concern about teachers learning to game the system. Many ranking systems work very well at first and then people learn to optimize against them. Just consider how many articles there are on how to improve your FICO score.

But the core ideas remain to find a way to measure teacher performance on some sort of low dimensional scale and to reduce regulation of private sector competitors. It is unclear that these approaches will get at the key issues, even if (due to an extraordinarily dysfunctional school system) they may produce some improvements in the Washington DC schools.

More on the SRI KIPP report

This is a fairly small case of researchers being (deliberately?) obtuse, but it's representative enough to merit a brief post.

First look at this passage from Wikipedia:

Some observers, such as the authors of The Charter School Dust-Up,^[5] say that KIPP's admission process self-screens for students who are both motivated and compliant, from similarly motivated and compliant—and supportive—families. Parents must commit to a required level of involvement, which rules out badly dysfunctional families.

You'll notice this is consistent with some points Joseph and I have been arguing for awhile about how selection and peer effects can affect a school.

Now pay close attention to the following from SRI:

KIPP schools’ higher-than-expected test score results draw both attention and claims that they “cherry-pick” high-achieving students from poor neighborhoods. This is the first report to closely scrutinize the praise and criticisms associated with KIPP, as well as key challenges facing Bay Area KIPP schools today.

In the three KIPP schools where they were able to draw comparisons, SRI researchers found that students with lower prior achievement on the CST were more likely to choose KIPP than higher-performing students from the same neighborhood, suggesting that, at least at these schools, cherry-picking does not occur.

Did you catch the shift? Like most areas, academic success is largely determined by attitude and work habit. This is particularly true in a school with extended hours of instruction and a longer year. There is every reason to believe that these traits are strongly selected for here, just as suggested by the Economic Policy Institute.

The Charter School Dust-Up came out from the EPI in 2005, three years before the SRI report, but the SRI researchers never directly acknowledge the EPI's conflicting findings. Instead they pull the researchers' version of the old school kid's trick of pretending to mishear an inconvenient question.

Calling Yves Smith

(Via Andrew Gelman) John Sides has good take-down of Steven Hayward's odd screed for the American Enterprise Institute's house journal. Here's a choice exchange.

From Hayward:

Of course, this is just the silly stuff. The real problem with academic political science is its insistence on attempting to emulate the empiricism of economics and other social sciences, such that the multiple regression analysis is considered about the only legitimate tool of the trade. Some regressions surely illuminate, or more often confound, a popular perception of the political world, and it is these findings Klein rightly points out. But, on the other hand, I have often taken a random article from the American Political Science Review, which resembles a mathematical journal on most of its pages, and asked students if they can envision this method providing the mathematical formula that will deliver peace in the Middle East. Even the dullest students usually grasp the point without difficulty.

And from Sides:

This is sort of bizarre. Let’s leave aside the notion that “multiple regression analysis” is the “only legitimate tool.” That’s the impression of someone who doesn’t read much political science. I’m more interested in Middle East peace. Here’s my question: if Hayward picks up the American Economic Review, does he envision that their mathematical formulas will produce global prosperity? That’s the standard to which he seems to hold academic research. If so, he should be disappointed by virtually the entire corpus of social science, and perhaps by a decent bit of the hard sciences as well. After all, there’s still that cancer thing.

I'm going to have to take Sides' side here (and not just for the wordplay), but what really struck me as amusing was the use (starting with the subtitle) of economics as the gold standard of empiricism. Is that really the way you want to play it give the current state of economics in general and macroeconomics in particular?

This is so funny it made me cry

From Dr. Girlfriend.

Is this not every bio-statistician’s worst nightmare?

They just don't care

I am sure that most people in the news media watch every episode of the Daily Show and the Colbert Report. Then they get up the next morning, go to work, and do their God-damnedest to drag American journalism to its lowest point since Benjamin Franklin Bache accused George Washington of collaborating with the British.

They know what they're doing. They just don't care.

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
Islamophobiapalooza
www.thedailyshow.com
Daily Show Full Episodes Political Humor Tea Party

So which one is it?

DeanDad brings this point up in a post today. It's not related to the main point of the post but it is certainly worth noting:

Hell, while we’re at it, let’s make a point of generating enough math teachers so that every state in the country can require four years of math in high school. Get the public K-12 system up to basic competence, and see what happens.

I think, at some level, we need to decide what the priorities in education are. Is there a shortage of competent math teachers? If so then a process in which we make it harder to become a teacher would seem to be counter-productive. After all, we are also grappling with real issues of potential reductions in the number of teachers due to budgetary constraints.

I think that failing to decide on these issues are at the heart of my concerns about education reform. Reformers point to schools like KIPP that have attrtion rates that make them infeasible as a national model. So that doesn't seem to be a way forward. But in the actual education system, we are discussing reducing resources and trying to compensate with higher quality.

Now, add in that the metrics used to evaluate schools have serious concerns (as pointed out by a wide variety of researchers) and it gets hard to see what the road forward looks like. Clearly, if we are reducing resources to education then we can't achieve this with a higher investment. We might gain some efficiency by breaking contracts with current teachers (over tenure and pensions) but such actions tend to increase costs in the long run.

So I think that the real thing that I want to see out of educational reform is specific proposals. Honestly, I suspect that a series of initiatives at the school district level (focusing on the issues in each area) might be the way to go. But I worry that the current approach seems far too focused on test scores and not on the actual process of education.

The appeal of bad ideas -- why do rational executives keep trying ideas that never work?

A while back my friend Brian (of Ultrasonic Remote) and I found ourselves on the subject of TV remakes, shows that kept the title and general premise of an old hit but rebooted it with a new cast. After a while a pattern became obvious, the few successes all had something in common: they were all science fiction or fantasy shows (and even there the odds were not good).

When we went out of that subgenre, we couldn't come up with a single success. Many of the shows were high profile affairs with talented casts (Kojak, Dragnet, the Fugitive) but we are still talking about a failure rate of pretty much 100%.

Eventually, some show will break the streak. Perhaps it will be Hawaii Five-O or the new Rockford Files (after all, how hard could it be to replace James Garner?), but this still begs the question, why do highly paid executives in a tremendously competitive field insist on trying things that have repeatedly and consistently failed?

(Acknowledging) Failure Is Not an Option

Here at OE, we have been spending a lot of time discussing the horrible student attrition rates in Bay area KIPP schools, but when I first came across the statistic, I hesitated before using it in a post. The number was so absolutely damning that I had to wonder about the source. Had some anti-charter school group gone through all the data and cherry-picked the worst thing they could find?

So I looked up the source of the statistic and found a veritable sales brochure for KIPP. The article opened with the headline:

"New Study Finds San Francisco Bay Area KIPP Students Outperform Peers"

followed by the subtitle:

"Combination of Key Features Contributes to Success, Provides Lessons for Other Public Schools"

The rest of the article continues in the same vein before finally getting to this:

As researchers analyzed the student achievement data and KIPP’s approach, they also identified challenges facing Bay Area KIPP schools, including high student attrition rates, teacher turnover, and low state and local funding. For example, 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade. Annual teacher turnover rates have ranged from 18 to 49 percent since 2003-04.

Putting aside the human cost paid by the students who fell into that sixty percent and their families (which is huge), the selection/attrition process described in this post results in a school filled with roughly the top quartile of hard-working, dedicated students. All but the most incompetent administrators will have spectacular results under those circumstances. The performance described here ("In most grades, Bay Area KIPP students make above-average progress compared with national norms, and four out of five KIPP schools outperform their host district.") is, if anything, on the low end of what we would expect yet it is written up in the most glowing terms imaginable.

When it comes to charter schools, there's a long tradition of bad news stories with positive or at least neutral headlines. Consider this article from the New York Times. The key paragraph was:

But for all their support and cultural cachet, the majority of the 5,000 or so charter schools nationwide appear to be no better, and in many cases worse, than local public schools when measured by achievement on standardized tests, according to experts citing years of research. Last year one of the most comprehensive studies, by researchers from Stanford University, found that fewer than one-fifth of charter schools nationally offered a better education than comparable local schools, almost half offered an equivalent education and more than a third, 37 percent, were “significantly worse.”

And what was the headline for this grim finding?

"Despite Push, Success at Charter Schools Is Mixed"

I genuinely believe that there are some wonderful charter schools out there and we can greatly improve education by building on these successes, but in order to do that we need to recognize and honestly admit the failures.

Up until now, that honesty has been in short supply.

Substituting metrics for judgement

This episode of This American Life concludes with a fascinating account of the manipulation of crime statistics. Drop by, take a listen, give them a buck.

If reformers are looking for an indictment of our educational system...

I'd like to put some finer shadings on Joseph's recent post. I think I speak for most former and current teachers when I say that my favorite part of teaching was teaching -- explaining concepts, helping students working through problems, convincing them that they can not only do math, they can actually be good at it.

Helping a student who has shown an interest in learning always takes a high priority, but at some point something has to give. When I was teaching I routinely worked fifty plus hours a week when you count grading and making lesson plans. Add being on call ten or more hours a week and you will eventually have to make a choice. You can blow off the kids, you can blow off the grading and lesson plans or you can find out how long you can go without seeing your family or getting over six hours sleep.

As far as I can tell, the underlying assumption of the KIPP business model is an unlimited supply of Erin Gruwells. In case you've forgotten, Ms. Gruwell was the... let's say, inspiration for the character played by Hillary Swank in the movie Freedom Writers. Ms. Gruwell has managed an exceptional career but as a teacher, she followed a very common trajectory. She came into the field with high expectations and tremendous energy, worked incredibly long hours at a fantastic pace, achieved some early successes then bailed in less than five years.

KIPP burns through these teachers at an extraordinary rate ("Annual teacher turnover rates have ranged from 18 to 49 percent since 2003-04"). This is possible in large part because there are less than twenty-seven thousand students in the KIPP system. By charter school standards, this is substantial but there are over sixty-million K through 12 children in the U.S.

A number of people think KIPP is a scalable model for the nation. Many of these people hold advanced degrees from some of America's most prestigious universities. If reformers are looking for an indictment of our educational system, I can't think of a more damning one.

Cost of Education

I was speaking with Mark about this post and an interesting point came up. Teachers already have long and structured hours (pre and post class supervision, class hours and lunch duty) plus work at home (marking -- which at the high school level is never finished during the workday). KIPP has teachers also available by cell phone. In the private sector it generally involves a pretty hefty pay increase to put an associate on a pager.

When we talk about the need to increase our investment in education, are we really prepared to increase either the number of teachers or the median salary of teachers? After all, we are currently talking about teacher layoffs. Perhaps Mark, who has more practical experience in this matter, can comment?

Reflections on the KIPP posts

I want to explicitly emphasize this point in Mark's last post:

One is the eighty/twenty rule. Some students take more time than others and, not surprisingly, the students who lower a school's test average and management metrics are the ones who consume the most time and resources.

In the light of the KIPP website (which he discusses here):

Having teachers available by cell phone after school for homework help

Now, let us consider two effects of this policy. One, the school day is a lot longer for a teacher than it used to be. Add in the rigorous extracurricular activities and the (seemingly 24 hour on call status) and at best you are creating an environment where investing more resources in teaching creates better outcomes.

But I don't think that this is going to be the main effect. If you consider Mark's eighty/twenty rule, you create enormous incentives for teachers to gently nudge out the student who calls for an hour and half to discuss homework every single evening. Even if the teachers at KIPP don't do this (a proposition of which I would be highly sceptical), I would be deeply concerned about what whether this approach was scalable.

It think it would be critical to determine if an approach was scalable before basing massive reforms off of it. I am happy for innovation and improvement in education to continue but I think that we should proceed carefully and based on the evidence. In education, for reasons that evade me, people seem to be very non-critical about the evidence.

Make that most children left behind

In an email, Joseph accused me of burying the lede when I discussed KIPP charter schools in an earlier post. The paragraph in question was this quote from Wikipedia:

"In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."

Administrators have long known that the simplest and most reliable way to improve a school's performance is by selection and attrition of the student body. This works in the obvious, direct ways -- if you drop the kids who disrupt class and/or can't master the material, test scores and classroom management metrics will go up -- but there are at least a couple of indirect effects that are as, or more, powerful.

One is the eighty/twenty rule. Some students take more time than others and, not surprisingly, the students who lower a school's test average and management metrics are the ones who consume the most time and resources.

Even more significant are peer effects. K through 12 students are particularly sensitive to perceived social norms. By selectively removing certain students from the population, you can easily create a high degree of conformity to an almost ideal set of behaviors and attitudes.

Most educators look at getting rid of students as a last resort. The prevailing attitude is that you are there to help all the kids, not just the easy ones. You will find exceptions, of course, like principals who are a little too eager to expel certain students or find ways to influence which students are assigned to other schools, but they are forced to work around rules that discourage this behavior.

For charter schools, though, these selective factors are built into the system. Here's the deal that college prep charter schools offer, if the students go through an involved application/induction process, take more difficult courses during a longer school day and do more homework, they will have a better chance at academic and professional success. These schools have automatically limited their pool of applicants to kids who consider academic and professional success both desirable and attainable and who come from supportive, involved families that are willing to make a real effort to give their children a chance to succeed.

(quick clarification: we are talking about selection processes that favor certain attitudes and behaviors. We are NOT talking about favoring students with high test scores. Many reporters and, God help us, researchers have failed to grasp this distinction.)

Charter schools segregate out a group of students who tend to be, to put it bluntly, easy to teach. This is not an entirely bad thing. Though there are concerns about this being a zero-sum-game, there is something to be said for making sure that every neighborhood has at least one educational bright spot.

It is, however, an entirely bad thing when dishonest or naive observers evaluate these schools without taking these systemic advantages into account and it is worse still when unscrupulous administrators try to build on those systemic advantages by cooking the data through selective attrition.

If you start with a student body made up almost entirely of kids who want to be in school, avoid gangs and graduate from college, who are supported by families with the same goals, you should expect relatively low attrition. When you see the opposite, you should certainly be suspicious. Even the most inept administrator can look good if you allow him or her to pick the most promising students out of an already select group.

The 'I' stands for 'Ironic'

Check out bullet two. (From the KIPP website)

At KIPP schools:

• Parents are encouraged to be involved and to contribute;
• Teachers have the freedom to innovate and never give up on a child;
• Students work hard and come to school ready to learn.

Update: While you're here, why not take a look at the next post

Rationality

From Econned page 148:

Greenspan was hardly alone in his dogmatic belief in the wisdom of leaving “free markets” to their own devices . . . Judge Frank Easterbrook and Daniel Fishel, then University of Chicago Law School dean, asserted in 1991: “[A] law against fraud is not an essential or necessarily important ingredient of securities markets.”

I find this line of thinking to be rather remarkable. The two classic libertarian roles for government are to prevent force and fraud. Markets can’t be efficient if information is missing or removed – in a world where identities can be re-invented then you simply can’t have a national (let alone) global market if fraud is a permissible business strategy. The only way to trust people with your money or in business deals would be a long history of relationships.

This line of thinking requires one to think that there are no information asymmetries in the real world. Furthermore, a rational agent permitted to engage in fraud might be willing to destroy their relationship for a large enough pay-off. Yves Smith gives an example of how this thinking is leads to odd conclusions:

Easterbrook went so far as blocking a plaintiff from presenting a case that argued that an auditor has assisted in a fraud. The judge claimed it would be “irrational” for the accountant to behave in that way, given his interest in preserving his reputation.

Of course, one could argue that all potential criminals have an interest in preserving their reputation. But one might very well risk their reputation for a sufficiently high pay-off – especially if one is highly unlikely to be prosecuted.

I do not think that this type of thinking is universal among “free market” advocates but it is extremely concerning that these arguments can be advanced without public mockery.

More from Yves Smith

An interesting point in Econned (pages 223-224):

One way to contain compensation is for the central bank to raise interest rates when inflation starts to build. The logic is that increasing unemployment will moderate pay pressures and also discourage business from giving employers pay increases in excess of productivity gains. Ironically, quite a few “free markets” supporters endorse this type of intervention to correct a perceived marker failure (labor having undue bargaining power) but reject a raft of others.

One of the hard things about free markets is that everybody seems to want to meddle with at least one small portion that they see as problematic. However, if you are not careful you can end up with unexpected consequences as markets respond to incentives in unexpected ways. This is not to say that inflation is a good thing (it isn’t) but rather to recognize that real markets have limitations and failures.

This example is particularly interesting as one of the great issues of the moment is how powerless workers have become in the face of tight credit markets.

Fishing expeditions

This has to be a bad idea. Chronic pain is a difficult condition to treat already with an extremely limited set of options; second guessing treatment decisions seems to be likely to lead to more suffering in the long run.

Changing statistical languages

Even when a current programming language has drawbacks, it can be hard to change to a more optimal language due to the investment in the current language. Here is a comment from Julien Cornebise:

But R is here and in everyday use, and the matter is more of making it worth using, to its full potential. I have no special attachment to R, but any breakthrough language that would not be entirely compatible with the massive library contributed over the years would be doomed to fail to pick-up the everyday statistician—and we’re talking here about far-fetched long-term moves. Sanitary breakthrough, but harder to make happen when such an anchor is here.

R is a pretty amazing language and, as a long term SAS user, I must admit that I am delighted by the graphics and the cool packages. Of course, the dark side of such a rich library is needing to learn about the reliability and limitations of all of the different packages.

H/t: Andrew Gelman

Strange Bedfellows

(where your intrepid blogger sides with Robert Samuelson, gives Ray Fisman a break and wonders what in the hell happened to Jonathan Chait)

You really have to read this Jonathan Chait column on Robert Samuelson, and I don't mean that in a good way. You have to read this to see for yourself how low the educational reform movement can drag even a writer as gifted as Chait.

There's too much here to cover in one post (I could do a page just on Chait's weird reaction to Samuelson's looks, a topic that I had never given any thought to up until now). I may take another pass at another section later but for now I'm going to limit myself to this particularly egregious bit:

How does Samuelson explain the existence of new charter schools that produce dramatically higher results among these lazy, no-good teenagers? He insists, "no one has yet discovered transformative changes in curriculum or pedagogy, especially for inner-city schools, that are (in business lingo) 'scalable.'" This is utterly false. The most prominent example is the Kipp schools, which have shown revolutionary improvements among poor, inner-city students and have rapidly expanded.

It is strange to see Chait take the pro-privatization side of the debate, stranger still to see him accuse critics of charter schools of having an anti-government bias*, but what pushes this into Rod Serling territory is the spectacle of having Chait, one of the most gifted bullshit detectors of the Twenty-first Century, rolling out the same sort of flawed argument that he has made a career out of dismantling.

In order to be viable, a reform has to improve on the existing system by a large enough margin to justify its implementation costs, but if you accept the metrics used by the reform movement, then you will have to conclude that charter schools do worse than public schools more often than they do better.**

So we have a major push to privatize government services which, after about two decades of testing have been shown to under-perform their traditional government-run alternatives. Rather than show why this statistic is misleading, Chait pulls out vague, anecdotal evidence of a single out-lier. Now, given the variability of the data, we would expect the top schools (or even chains) to do pretty well. That alone rebuts Chait's point, but it gets worse. Self-selection, peer effects and selective attrition*** all artificially inflate KIPP's results. When you take these factors into account, it's hard to make a compelling statistical case that even the best charter schools are outperforming public schools (though the second footnote still applies).

At the risk of over-emphasizing, this is Jonathan -- freaking -- Chait we're talking about, a writer known for his truly exceptional gift for constructing logical arguments and, more importantly, spotting the fallacies in the arguments of others. Under normal conditions, Chait would never fall for a badly presented argument-by-anomally, let alone make one, just as, under normal circumstances, a confrontation between Samuelson and Chait would result in little pieces of the former being scraped off of the walls of the Washington Post.

But Chait loses this confrontation decisively. From his ad hominem opening to his factually challenged close he fails to score a single point. And this is far from the only example of this odd reform-specific impairment affecting otherwise accomplished writers. OE has spilled endless pixels on the reform-related lapses, both statistical and rhetorical, of smart, serious, dedicated people like Chait, Seyward Darby and, of course, Ray Fisman (just do a keyword search). None of these people would normally produce the kind of work we've cataloged here. None of them would normally ignore the defection of one of the founding members of the reform movement. None of these people would normally feel comfortable dismissing without comment contradictory findings from EPI, Donald Rubin and the Rand Institute.

David Warsh has aptly made the following comparison:

Remember the recipe for a policy disaster? Start with a handful of policy intellectuals confronting a stubborn problem, in love with a Big Idea. Fold in a bunch of ambitious Ivy League kids who don’t speak the local language. Churn up enthusiasm for the program in the gullible national press – and get ready for a decade of really bad news. Take a look at David Halberstam’s Vietnam classic The Best and the Brightest, if you need to refresh your memory. Or just think back on the run-up to the war in Iraq.

but along with Halberstam, it might be time to brush off our copies of Cialdini's Influence.

From a data standpoint, the past few years have been rough on the reform movement. Charter schools have been shown to be more likely to under-perform than to outperform. Joel Klein's spectacular record turned out to be the product of creative accounting (New York City schools have actually done much worse than the rest of the state). Findings contradicting the fundamental tenets of the movement accumulated. Major figures in research (Rubin) and education (Ravitch) have publicly questioned the viability of proposed reforms.

As Cialdini lays out in great detail, when you challenge people's deeply held beliefs with convincing evidence, you usually get one of two responses. Sometimes you will actually manage to win them over. More often, though, they will dig in, embrace their beliefs more firmly and find new ways to justify them.

I think it's safe to say we don't have response number one.




* Almost all of the major tenets of the modern reform can be traced back to the Reagan era and were closely associated with the initiatives described in Franks' The Wrecking Crew.

** Ironically, if you consider the intellectual framework of the reform movement to be flawed and overly simplistic, you can actually make a much better case for charter schools.

*** From Wikipedia: "In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."

Non-compete agreeements

Felix Salmon is discussing the lawsuit over Mark Hurd accepting a job at Oracle. The crux of the argument seems to be concerns that Mr. Hurd might reveal confidential information as part of his new job. Fortunately, for him, the state of California takes a very dim view of non-compete agreements. From commenter Vania, in the comments to Mr. Salmon's post:

The covenant not to compete, as written, is simply unenforceable under California Business & Professions Code Section 16600:

“16600. Except as provided in this chapter, every contract by which anyone is restrained from engaging in a lawful profession, trade, or business of any kind is to that extent void.”

Now I have a generally dim view of non-compete agreements, so I think that this law is good sense. While I don't have any issue with a CEO having more restrictive rules, given the level of compensation that they are given, I am dubious that these agreements are given only at the CEO level. I know that I was under one, once, in a very junior position at a firm.

The key issue is that I am unsure of how easily one can freely consent to such an agreement in the midst of a dismissal. The person has just had their life turned upside down and likely lost a crucial income stream. The company has had time to prepare the exit package and carefully optimize it for their interests. They have had lawyers look it over and had HR vet the relevant policies. I am unclear that these structural differences in information can be overcome nor do I really see the "sign this or get no severance" as being a real choice for people who have had no ability to assess their options.

At an even more fundamental level, I am unclear why these sorts of agreements don't violate our norms of a free market. How can there be a bigger barrier to free economic activity than a pledge for workers not to sell their skills? We are already worried about issued with implied compensation, does this not count as a hidden cost that is not declared up front?

At best one might argue for contract law and the ability of people/organizations to enter into agreements. But the santicity of contract law seems to be under attack when it favors the worker (consider tenure and defined benefit pension plans). Why do the same arguments about net social good not apply here?

Vintage of Educational Reform

I just wanted to make a point in the education debate in passing. It is well worth re-reading Mark's post on his experience with these same reforms:

The trouble is that almost none of the people using the term 'reform' are actually suggesting any reforms. Most of the proposals that have been put forward are simply continuations or extensions of the same failed policies and questionable theories that have been coming out of schools of education for years, if not decades.

In 1994, RAND did a policy brief on the use of standardized scores in education. Look at the conclusions (from 16 years ago):

Research has not been able to pinpoint the effects of the noneducational influences. Nevertheless, people have misused test-score data in the debate to give education a "bad rap." Koretz lists three broad, overlapping kinds of misuse that should be avoided in honest, future debate:

1.Simplistic interpretations of performance trends: These trends should not be taken at face value, ignoring the various factors that influence them: for example, demographic changes in test takers or inflation of scores caused by test-based accountability.

2.Unsupported "evaluations" of schooling: Simple aggregate scores are not a sufficient basis for evaluating education--unless they provide enough information to rule out noneducational influences on performance. Most test-score databases do not offer that kind of information.

3.A reductionist view of education: Koretz notes that it may be "trite" but it is true that education is a "complex mix of successes and failures . . . what works in one context or for one group of students may fail for another." Unfortunately, that truism is often ignored. For example, in the early 1980s, when people were reasonably concerned about falling aggregate test scores, they asked for wholesale changes in policies, without first asking which policies most needed changing or which students or schools most needed new policies.

Maybe we should think twice before declaring that drops or stagnation in scores are necessarily evidence that education is failing? As for the performance gains in charter schools, Mark has been after that issue from the beginning of this discussion here at OE.

It's worth keeping this in mind when evaluating calls for massive reforms on the basis of standardized test scores alone.

Interesting thought

From Grognardia:

Indeed, if there's one "problem" with reading a book like Galactic Patrol nowadays it's that, as the wellspring of so much that came after, it appears trite and unoriginal when, in point of historical fact, everything else that followed it is what's trite and unoriginal. The "Lensmen" series is big and bold and, while I'd never argue that it's scientific speculations hold much water (though, to be fair, many of its ideas were based on the science of its day), it's nevertheless a fun read. There can be no doubt why it exerted such a profound influence on the imaginations of later authors in the genre.

I think that this is a very insightful point that brings up a good point when reading key historical works. Good ideas spawn a lot of imitators and some of these will be better (in terms of some aspects) than the original works. Just think of how many of the imitators of Lord of the Rings are better at pacing the plot! But who among them has such a rich and unique vision of an alternate world?

But these are the works that change entire genres of fiction.

Education in Canada

Another interesting point from Worthwhile Canadian Initiative:

The demand for French immersion education in Vancouver so far outstrips the supply that the school board allocates places by lottery.

But why? Is it because French is a useful employment skill? Because learning to speak French makes you a better person? Or is it because parents know intuitively what economists can show econometrically: peer effects matter. Being with high achieving peers raises a student's own achievement level.

Consider this point quoted in that article:

If students with special needs were equally distributed among all classes, each teacher would on average have 3.4 students with special needs. However, in schools that have early immersion programs, the average in core English programs is about 5.7 students.

I think that it is quite possible that these results could be applied to charter schools with long waiting lists in the United States. Why does this matter?

Because the use of lotteries could otherwise be considered to be a form of randomization. But it seems odd that being taught in a second language would result in better educational outcomes, per se. Which suggests that Frances Woolley has a point that peer effects really do matter and we should consider this when evaluating outcomes in US charter schools.

Update: Of course, I forget to mention that Mark brought up this exact point at the beginning of our foray into education.

Statistical significance (a never-ending series)

Andrew Gelman has a post on a mis-definition of the p-value. I want to focus on another aspect of the quote:

Despite the myriad rules and procedures of science, some research findings are pure flukes. Perhaps you're testing a new drug, and by chance alone, a large number of people spontaneously get better. The better your study is conducted, the lower the chance that your result was a fluke - but still, there is always a certain probability that it was.

Statistical significance testing gives you an idea of what this probability is.

This is not only an incorrect definition of the p-value but it also appears to be ignoring the possibility of bias and/or confounding. Even in a randomized drug trial (and drug trials are explicitly being used as an example), it is possible to induce selection bias due to non-random loss to follow-up in any non-trivial study. After all, many drugs are such that the participants can guess their exposure status (all analgesics have this unfortunate property) and this can lead to a differential study completion rate among some sub-groups. For some outcomes (all-cause mortality), complete ascertainment can be done using an intention to treat approach to analysis. But that typically induces a uniform bias towards the null.

I am always uncomfortable with how these strong and unverifiable assumptions are glossed over in popular accounts of pharmacoepidemiology.

Another troubling study

From the EPI paper:

Because of the range of influences on student learning, many studies have confirmed that estimates of teacher effectiveness are highly unstable. One study examining two consecutive years of data showed, for example, that across five large urban districts, among teachers who were ranked in the bottom 20% of effectiveness in the first year, fewer than a third were in that bottom group the next year, and another third moved all the way up to the top 40%. There was similar movement for teachers who were highly ranked in the first year. Among those who were ranked in the top 20% in the first year, only a third were similarly ranked a year later, while a comparable proportion had moved to the bottom 40%.

What's really amazing here is that the authors of the fire-the-bottom-80-percent paper actually cite other work by Timothy Sass and yet manage to overlook this.

EPI Briefing Paper -- Problems with the Use of Student Test Scores to Evaluate Teachers

In terms of education reform, this is probably the biggest story to come over the wires in a long time:

While there are good reasons for concern about the current system of teacher evaluation, there are also good reasons to be concerned about claims that measuring teachers’ effectiveness largely by student test scores will lead to improved student achievement. If new laws or policies specifically require that teachers be fired if their students’ test scores do not rise by a certain amount, then more teachers might well be terminated than is now the case. But there is not strong evidence to indicate either that the departing teachers would actually be the weakest teachers, or that the departing teachers would be replaced by more effective ones. There is also little or no evidence for the claim that teachers will be more motivated to improve student learning if teachers are evaluated or monetarily rewarded for student test score gains.

Read the paper here, then take a look at Kenneth J. Bernstein's detailed analysis and Joseph's brief explanation of why we should listen to Donald Rubin.

The principal effect -- a footnote to the last post

When it comes to education reform, you can't just refer to the elephant in the room. It's pretty much elephants everywhere you look. There is hardly an aspect of the discussion where reformers don't have to ignore some obvious concern or objection.

The elephant of the moment is the effect that principals and other administrators have on the quality of schools. Anyone who has taught K through 12 can attest to the tremendous difference between teaching in a well-run and a badly-run school. Even the most experienced teacher will find it easier to manage classes, cover material, and keep students focused. All of those things help keep test scores up, as does the lower rate of burn out. For new teachers, the difference is even more dramatic.

On top of administrator quality, there is also the question of compatibility. In addition to facing all the normal managerial issues. teacher and and principal have to have compatible educational philosophies.

As we've mentioned more than once on this site, educational data is a thicket of confounding and aliasing issues. That thicket is particularly dense when you start looking at teachers and principals and, given the concerns we have about the research measuring the impact of teachers on test scores, I very much doubt we will ever know where the teacher effect stops and the principal effect starts.

Addiction

I think that it is easy to understate how hard it can be to eliminate an addiction, even one where we know about the potential harms (i.e. smoking). I found this passage really interesting:

One time I had just enough money to put in the electric meter or buy a packet of cigarettes. There I was sat in the dark smoking like a chump trying to comfort myself with the fact that it was not crack.

These sorts of tales really make me ponder whether we should focus more on "harm reduction" and if elimination may not be a quixotic pursuit.

Oh, Canada -- another interesting omission in "Clean Out Your Desk"

We're back with our ongoing coverage of Ray Fisman's recent article in Slate which ran with the provocative tagline "Is firing (a lot of) teachers the only way to improve public schools?" (notice that he didn't say "a way" or "the best way").

If you tuned in late, here's what you need to know:

Dr. Fisman starts by discussing a presidential commission report from the early Eighties that said the damage done by our poor educational system was comparable to an act of war. This somewhat apocalyptic language has since become a staple of the reform movement. It grabs the attention, justifies big, expensive, untried steps and sets up a false dichotomy between action and inaction.

The proceedings are then handed over to Joel Klein. Klein builds on the verge-of-disaster theme by invoking the United States' low ranking on the Organization for Economic Co-operation and Development's PISA tests. I've commented at some length on the implications of citing PISA while completely ignoring the better-established and well-respected TIMMS even when the discussion shifted to elementary schools where the TIMMS scores would seem to be far more relevant. (The term cherry-picking did come up.)

For now, though, let's grant Chancellor Klein and Dr. Fisman the benefit of the doubt. Let's say we accept the premise that OECD's PISA rankings are such a good and reliable measure of the state of a nation's schools that we don't even need to look at other metrics. We'll even stipulate for the sake of argument that a bad PISA ranking is sufficient grounds for radical measures. With all of these conditions in place, take close look at the next part of Dr. Fisman's article:

What could turn things around? At a recent event that I organized at the Columbia Business School, Klein opened with his harsh assessment of the situation, and researchers offered some stark options for getting American education back on track. We could find drastically better ways of training teachers or improve our hiring practices so we're bringing aboard better teachers in the first place. Barring these improvements, the only option left is firing low-performing teachers—who have traditionally had lifetime tenure—en masse.

The emphasis on better teachers—through training, selection, or dismissal—comes from the very consistent finding that improving faculty is one of the best, most reliable ways to improve schools. If the person standing at the front of the classroom has raised the test scores of students he's taught before, he's likely to do so again.

But how do you get good teachers in the classroom? Unfortunately, it turns out that most evidence points toward great instructors being born, not made. National board certification may help a bit, a master's degree in education not at all. It's also difficult to pick out the best teachers based on a résumé or even a sample lesson. It takes a year or so before evaluators (and even teachers themselves) know who is really good at getting kids to learn, and few qualifications are at all correlated with teaching ability. Candidates with degrees from prestigious colleges—the type where Teach for America does much of its recruiting—do a bit better, but not much.

Here's the gist of Dr. Fisman's premise:

1. According to PISA (the test that trumps all other tests) the state of U.S. education is dire;

2. We need to improve the quality of our teachers "through training, selection, or dismissal";

3. So far, no one has found a way to make training or selection work.

If we want education to do well we might just have to start firing teachers en masse, and by "do well," we mean outscore other countries, which raises the question, "How do other countries find all of those natural teachers?"

Of course, comparing educational systems of different countries can be tricky but we should at least be able to look at Canada. It's a fairly large industrialized country. Not that different economically. Very similar culturally with a comparable K through 12 educational system that has to deal with English as a second language (huge immigrant population), relies on roughly the same type of teacher training/certification that we use and continues to pull teachers in with promises of good job security.

In terms of this discussion, the biggest difference between the two countries could well be Canada's somewhat reactionary approach to reform (for example, only one province, Alberta, allows public charter schools). With such limited school choice and no real attempt to clean out the deadwood from behind the podium, the Canadian educational system looks a lot like the American system before the reform movement.

And how is Canada doing on the PISA math test?

From Measuring up : Canadian Results of the OECD PISA Study:

One way to summarize student performance and to compare the relative standing of countries is by examining their average test scores. However, simply ranking countries based on their average scores can be misleading because there is a margin of error associated with each score. As discussed in Chapter 1, when interpreting average performances, only those differences between countries that are statistically significant should be taken into account. Table 2.1 shows the countries that performed significantly better than or the same as Canada in reading and mathematics. The averages of the students in all of the remaining countries were significantly below those of Canada. Overall, Canadian students performed well. Among the countries that participated in PISA 2006, only Korea, Finland and Hong Kong-China performed better than Canada in reading and mathematics. Additionally Chinese Taipei performed better than Canada in mathematics.

That puts them in the top ten (in science they were in the top three). Now let's review the United States' performance (quoting Dr. Fisman):

Despite nearly doubling per capita spending on education over the past few decades, American 15-year olds fared dismally in standardized math tests given in 2000, placing 18th out of 27 member countries in the Organization for Economic Co-operation and Development. Six years later, the U.S. had slipped to 25th out of 30.

How do we reconcile these facts with Dr. Fisman's argument? As far as I can see, there are only four possibilities (if I've missing some please click the comment button and let me know):

1. Though PISA is a useful test, international PISA ranking may not be a sufficient measure of a country's school system;

2. Teacher quality is not a major driver of national educational performance;*

3. Teachers are made, not born. i.e. it is possible to train people to be good teachers;

4. Canada just got lucky and beat the odds hundreds of thousands of times.

If this were a PISA question, I hope no one would pick number four.



* This is really is a topic for another post, but I would expect the administrator effect to overwhelm the teacher effect. Perhaps Dr. Fisman is going to follow up with a Slate article on firing administrators who produce lackluster test performance.