Showing posts with label TIMSS. Show all posts
Showing posts with label TIMSS. Show all posts

Monday, September 27, 2010

Does Seyward Darby think Seyward Darby should be fired?

I don't. I have always believed that firing is a last resort, but Darby has certainly gone on the record as being for firing incompetent performers even when the metrics for measuring competence are unreliable and the firings would cause severe damage to the economy. Given the quality of her reporting on education, it's difficult to believe she's not currently lobbying TNR to get herself fired for things like this:
But the real star of the show was Waiting for Superman, the much-hyped documentary about school reform that opens nationwide this week. Gregory started the program with a clip from the movie that shows how poorly we rank, education-wise, against other developed countries.
Does the clip really show how poorly we're doing? Let's roll the tape:
Since the 1970s, U.S. schools have failed to keep pace with the rest of the world. Among 30 developed countries, we ranked 25th in math and 21st in science. The top 5 percent of our students, our very best, ranked 23rd out of 29 developed countries. In almost every category, we've fallen behind.
If you've been following OE, you recognize these oft-quoted numbers as coming from the PISA test which can't possibly support the first sentence since it was first administered in 2000.

It would seem that Seyward Darby doesn't know that.

She also doesn't seem to know that the older, better-established TIMSS test has us doing fairly well internationally. Nor is she apparently aware that using PISA to argue for the standard slate of reforms is problematic since at least one of the highest scoring countries (Canada) has adopted pretty much the opposite approach.

We can let David Gregory off with a warning -- this isn't his beat -- but Darby is the education specialist for one of America's best and most respected publications. There's no way for the New Republic to justify keeping an education reporter who can't spot obvious distortions involving one of the two best known measures of international academic performance.

I would suggest immediate reassignment. If Darby would like to argue for her own dismissal, I'd be happy to debate the issue.

[update: you can find some more thoughts on TNR's education reporting here.]

Wednesday, August 25, 2010

The Old Shell Game -- Why you have to keep your eye on Ray Fisman (and no, we're not quite through with the second paragraph)

Ray Fisman's Slate article "Clean Out Your Desk" is something of a greatest hits of the education reform movement, covering most of the standard arguments and carefully selected statistics that are brought up again and again by the advocates. It is, therefore, fitting that FIsman gets no further than the second paragraph before getting to this:
New York City Schools Chancellor Joel Klein often quotes the commission before discussing how U.S. schools have fared since it issued its report. Despite nearly doubling per capita spending on education over the past few decades, American 15-year olds fared dismally in standardized math tests given in 2000, placing 18th out of 27 member countries in the Organization for Economic Co-operation and Development. Six years later, the U.S. had slipped to 25th out of 30. If we've been fighting against mediocrity in education since 1983, it's been a losing battle.*
The OECD tests are the book of Revelations of the education reform movement, the great ominous portent to be invoked in the presence of critics and non-believers. Putting aside questions of the validity and utility of this test (perhaps for another post if my stamina holds out), we would certainly like to be in the top ten rather than the bottom.

But before we concede this one, lets pull out our well-thumbed copy of Huff and take one more look. Whenever one side in a complex debate keeps pulling out one particular statistic, you should always take a moment and check for cherry-picking.

Is Fisman distorting the data by being overly selective when picking statistics to bolster his case? Yes, and he's doing it in an egregious way.

Take a look at at the Trends in International Mathematics and Science Study. Here's a passage from the executive summary from the National Center for Education Statistics:

In 2007, the average mathematics scores of both U.S. fourth-graders (529) and eighth-graders (508) were higher than the TIMSS scale average (500 at both grades). The average U.S. fourth-grade mathematics score was higher than those of students in 23 of the 35 other countries, lower than those in 8 countries (all located in Asia or Europe), and not measurably different from those in the remaining 4 countries. At eighth grade, the average U.S. mathematics score was higher than those of students in 37 of the 47 other countries, lower than those in 5 countries (all of them located in Asia), and not measurably different from those in the other 5 countries.
We could spend some time in the statistical weeds and talk about the methodology of TIMSS vs. OECD's PISA. TIMSS is the better established and arguably better credentialed, but both are serious efforts mounted by major international organizations and it would be difficult to justify leaving either out of the discussion.

If Fisman had limited his focus to the education of high school students and simply ignored the data involving earlier grades, we would have ordinary misdemeanor-level cherry-picking. Not the most ethical of behavior, but the sort of thing most of us do from time to time.

But Fisman does something far more dishonest; he quietly shifts the subject to teachers in general and often to elementary teachers in particular (take a good look at the study that's at the center of Fisman's article).

This means that, when you strip away the obfuscation, you get the following argument.

1. The best metrics for tracking American education are international rankings on math tests;

2. The best way of improving America's education system is fire massive numbers of teachers, including those in areas where we are doing well on international rankings on math tests.

The bad news here is that we have a long way to go to make it through Fisman's article and it doesn't get much better, but the good news is that we're through with the second paragraph.




* Fought, for the most part with Klein and Fisman's battle plan but we've already covered that.