Does Seyward Darby think Seyward Darby should be fired?

I don't. I have always believed that firing is a last resort, but Darby has certainly gone on the record as being for firing incompetent performers even when the metrics for measuring competence are unreliable and the firings would cause severe damage to the economy. Given the quality of her reporting on education, it's difficult to believe she's not currently lobbying TNR to get herself fired for things like this:

But the real star of the show was Waiting for Superman, the much-hyped documentary about school reform that opens nationwide this week. Gregory started the program with a clip from the movie that shows how poorly we rank, education-wise, against other developed countries.

Does the clip really show how poorly we're doing? Let's roll the tape:

Since the 1970s, U.S. schools have failed to keep pace with the rest of the world. Among 30 developed countries, we ranked 25th in math and 21st in science. The top 5 percent of our students, our very best, ranked 23rd out of 29 developed countries. In almost every category, we've fallen behind.

If you've been following OE, you recognize these oft-quoted numbers as coming from the PISA test which can't possibly support the first sentence since it was first administered in 2000.

It would seem that Seyward Darby doesn't know that.

She also doesn't seem to know that the older, better-established TIMSS test has us doing fairly well internationally. Nor is she apparently aware that using PISA to argue for the standard slate of reforms is problematic since at least one of the highest scoring countries (Canada) has adopted pretty much the opposite approach.

We can let David Gregory off with a warning -- this isn't his beat -- but Darby is the education specialist for one of America's best and most respected publications. There's no way for the New Republic to justify keeping an education reporter who can't spot obvious distortions involving one of the two best known measures of international academic performance.

I would suggest immediate reassignment. If Darby would like to argue for her own dismissal, I'd be happy to debate the issue.

[update: you can find some more thoughts on TNR's education reporting here.]

My first thought on seeing McKinsey's “Closing the Talent Gap”

Where's Canada?

Here's the PDF of the report via this post from Matthew Yglesias. It just crossed my desktop and I have to be on the road in about five minutes. I've just had time to skim the report so I may be missing the obvious but the absence of our northern neighbor strikes me as strange, particularly given the report's use of PISA data.

Oh, Canada -- another interesting omission in "Clean Out Your Desk"

We're back with our ongoing coverage of Ray Fisman's recent article in Slate which ran with the provocative tagline "Is firing (a lot of) teachers the only way to improve public schools?" (notice that he didn't say "a way" or "the best way").

If you tuned in late, here's what you need to know:

Dr. Fisman starts by discussing a presidential commission report from the early Eighties that said the damage done by our poor educational system was comparable to an act of war. This somewhat apocalyptic language has since become a staple of the reform movement. It grabs the attention, justifies big, expensive, untried steps and sets up a false dichotomy between action and inaction.

The proceedings are then handed over to Joel Klein. Klein builds on the verge-of-disaster theme by invoking the United States' low ranking on the Organization for Economic Co-operation and Development's PISA tests. I've commented at some length on the implications of citing PISA while completely ignoring the better-established and well-respected TIMMS even when the discussion shifted to elementary schools where the TIMMS scores would seem to be far more relevant. (The term cherry-picking did come up.)

For now, though, let's grant Chancellor Klein and Dr. Fisman the benefit of the doubt. Let's say we accept the premise that OECD's PISA rankings are such a good and reliable measure of the state of a nation's schools that we don't even need to look at other metrics. We'll even stipulate for the sake of argument that a bad PISA ranking is sufficient grounds for radical measures. With all of these conditions in place, take close look at the next part of Dr. Fisman's article:

What could turn things around? At a recent event that I organized at the Columbia Business School, Klein opened with his harsh assessment of the situation, and researchers offered some stark options for getting American education back on track. We could find drastically better ways of training teachers or improve our hiring practices so we're bringing aboard better teachers in the first place. Barring these improvements, the only option left is firing low-performing teachers—who have traditionally had lifetime tenure—en masse.

The emphasis on better teachers—through training, selection, or dismissal—comes from the very consistent finding that improving faculty is one of the best, most reliable ways to improve schools. If the person standing at the front of the classroom has raised the test scores of students he's taught before, he's likely to do so again.

But how do you get good teachers in the classroom? Unfortunately, it turns out that most evidence points toward great instructors being born, not made. National board certification may help a bit, a master's degree in education not at all. It's also difficult to pick out the best teachers based on a résumé or even a sample lesson. It takes a year or so before evaluators (and even teachers themselves) know who is really good at getting kids to learn, and few qualifications are at all correlated with teaching ability. Candidates with degrees from prestigious colleges—the type where Teach for America does much of its recruiting—do a bit better, but not much.

Here's the gist of Dr. Fisman's premise:

1. According to PISA (the test that trumps all other tests) the state of U.S. education is dire;

2. We need to improve the quality of our teachers "through training, selection, or dismissal";

3. So far, no one has found a way to make training or selection work.

If we want education to do well we might just have to start firing teachers en masse, and by "do well," we mean outscore other countries, which raises the question, "How do other countries find all of those natural teachers?"

Of course, comparing educational systems of different countries can be tricky but we should at least be able to look at Canada. It's a fairly large industrialized country. Not that different economically. Very similar culturally with a comparable K through 12 educational system that has to deal with English as a second language (huge immigrant population), relies on roughly the same type of teacher training/certification that we use and continues to pull teachers in with promises of good job security.

In terms of this discussion, the biggest difference between the two countries could well be Canada's somewhat reactionary approach to reform (for example, only one province, Alberta, allows public charter schools). With such limited school choice and no real attempt to clean out the deadwood from behind the podium, the Canadian educational system looks a lot like the American system before the reform movement.

And how is Canada doing on the PISA math test?

From Measuring up : Canadian Results of the OECD PISA Study:

One way to summarize student performance and to compare the relative standing of countries is by examining their average test scores. However, simply ranking countries based on their average scores can be misleading because there is a margin of error associated with each score. As discussed in Chapter 1, when interpreting average performances, only those differences between countries that are statistically significant should be taken into account. Table 2.1 shows the countries that performed significantly better than or the same as Canada in reading and mathematics. The averages of the students in all of the remaining countries were significantly below those of Canada. Overall, Canadian students performed well. Among the countries that participated in PISA 2006, only Korea, Finland and Hong Kong-China performed better than Canada in reading and mathematics. Additionally Chinese Taipei performed better than Canada in mathematics.

That puts them in the top ten (in science they were in the top three). Now let's review the United States' performance (quoting Dr. Fisman):

Despite nearly doubling per capita spending on education over the past few decades, American 15-year olds fared dismally in standardized math tests given in 2000, placing 18th out of 27 member countries in the Organization for Economic Co-operation and Development. Six years later, the U.S. had slipped to 25th out of 30.

How do we reconcile these facts with Dr. Fisman's argument? As far as I can see, there are only four possibilities (if I've missing some please click the comment button and let me know):

1. Though PISA is a useful test, international PISA ranking may not be a sufficient measure of a country's school system;

2. Teacher quality is not a major driver of national educational performance;*

3. Teachers are made, not born. i.e. it is possible to train people to be good teachers;

4. Canada just got lucky and beat the odds hundreds of thousands of times.

If this were a PISA question, I hope no one would pick number four.



* This is really is a topic for another post, but I would expect the administrator effect to overwhelm the teacher effect. Perhaps Dr. Fisman is going to follow up with a Slate article on firing administrators who produce lackluster test performance.

The Old Shell Game -- Why you have to keep your eye on Ray Fisman (and no, we're not quite through with the second paragraph)

Ray Fisman's Slate article "Clean Out Your Desk" is something of a greatest hits of the education reform movement, covering most of the standard arguments and carefully selected statistics that are brought up again and again by the advocates. It is, therefore, fitting that FIsman gets no further than the second paragraph before getting to this:

New York City Schools Chancellor Joel Klein often quotes the commission before discussing how U.S. schools have fared since it issued its report. Despite nearly doubling per capita spending on education over the past few decades, American 15-year olds fared dismally in standardized math tests given in 2000, placing 18th out of 27 member countries in the Organization for Economic Co-operation and Development. Six years later, the U.S. had slipped to 25th out of 30. If we've been fighting against mediocrity in education since 1983, it's been a losing battle.*

The OECD tests are the book of Revelations of the education reform movement, the great ominous portent to be invoked in the presence of critics and non-believers. Putting aside questions of the validity and utility of this test (perhaps for another post if my stamina holds out), we would certainly like to be in the top ten rather than the bottom.

But before we concede this one, lets pull out our well-thumbed copy of Huff and take one more look. Whenever one side in a complex debate keeps pulling out one particular statistic, you should always take a moment and check for cherry-picking.

Is Fisman distorting the data by being overly selective when picking statistics to bolster his case? Yes, and he's doing it in an egregious way.

Take a look at at the Trends in International Mathematics and Science Study. Here's a passage from the executive summary from the National Center for Education Statistics:

In 2007, the average mathematics scores of both U.S. fourth-graders (529) and eighth-graders (508) were higher than the TIMSS scale average (500 at both grades). The average U.S. fourth-grade mathematics score was higher than those of students in 23 of the 35 other countries, lower than those in 8 countries (all located in Asia or Europe), and not measurably different from those in the remaining 4 countries. At eighth grade, the average U.S. mathematics score was higher than those of students in 37 of the 47 other countries, lower than those in 5 countries (all of them located in Asia), and not measurably different from those in the other 5 countries.

We could spend some time in the statistical weeds and talk about the methodology of TIMSS vs. OECD's PISA. TIMSS is the better established and arguably better credentialed, but both are serious efforts mounted by major international organizations and it would be difficult to justify leaving either out of the discussion.

If Fisman had limited his focus to the education of high school students and simply ignored the data involving earlier grades, we would have ordinary misdemeanor-level cherry-picking. Not the most ethical of behavior, but the sort of thing most of us do from time to time.

But Fisman does something far more dishonest; he quietly shifts the subject to teachers in general and often to elementary teachers in particular (take a good look at the study that's at the center of Fisman's article).

This means that, when you strip away the obfuscation, you get the following argument.

1. The best metrics for tracking American education are international rankings on math tests;

2. The best way of improving America's education system is fire massive numbers of teachers, including those in areas where we are doing well on international rankings on math tests.

The bad news here is that we have a long way to go to make it through Fisman's article and it doesn't get much better, but the good news is that we're through with the second paragraph.




* Fought, for the most part with Klein and Fisman's battle plan but we've already covered that.