Matt Yglesias -- Defending the indefensible

[I've been off line most of the week so instead of participating in the discussion on the Students First report card, I am, with apologies for the length, putting down my reaction in one big, ugly lump]

At some point in the past year, it became impossible to mount a serious defense of Paul Ryan. There had always been cracks in the facade -- numbers that didn't add up, unlikely claims, extremist quotes -- but most of these could be ignored and those that couldn't were invariably excused that Ryan was sincere, he was a serious budget guy and he was getting us to discuss important policy questions.

Eventually though, the discrepancies started to accumulate, and by the time we got the specifics (or non-specifics) of the Ryan budget and the close scrutiny of the campaign, the standard excuses simply weren't sufficient. This left a large number of journalists with a difficult choice: distance themselves from a politician they had invested great emotional and reputational capital in; or invest still more in increasingly strained defenses. The most memorable example of the first was William Saletan's amusing break-up letter. The most embarrassing example of the second probably comes from James Stewart.

In many ways, Michelle Rhee has occupied a Ryan-like niche in the education. Both started out as camera-friendly media darlings with highly marketable bipartisan appeal and reputations as serious problem-solvers. In both cases there were, from the beginning, troubling details that undercut these reputations but at the time these details never got much traction. As with Ryan and fiscal responsibility, criticizing Rhee was often read as indifference towards the education gap.

But, as they did with Ryan, nagging questions started to accumulate. There were incidents that seemed to show Rhee abusing her authority. There were questions about cheating under her watch. There was increasingly pointed anti-teacher rhetoric. There was the aggressive pursuit of self-advancement.  At each stage, more of Rhee's liberal supporters started getting uncomfortable.

For many, such as the New Republic's Seyward Darby, the tipping point came when Rhee partnered with Florida's Rick Scott. Before Scott, Rhee's liberal supporters had taken the position had been that she was tough on teachers, but reluctantly, and only because it was necessary in order to improve education. With Scott these outcomes were reversed. He was willing to pursue a "reform" agenda if it hurt a faction he saw as hostile.

The alliance with Scott and similar figures alienated some supporters on the left, but it still allowed the possibility that Rhee was acting in good faith. With the release of the Students First state report card, even that is gone. There is not even a pretense that this is about anything other than promoting Rhee and her allies. The Washington Post had a good summary.

In Rhee’s grading system, the D.C. school system that is implementing the reforms she instituted got a higher grade than the states of Maryland and Virginia — which consistently are at or near the top of lists of high-performing states — and Virginia. Maryland got a D-plus. Virginia got a D-minus. The District? The urban system with the highest achievement gap in the country? It got a C-plus. 

The states that got the highest score handed out — a B minus — were Florida and Louisiana. No surprise there.

Florida’s reform efforts were spearheaded more than a decade ago by then-Gov. Jeb Bush, who was the national leader in these kinds of reforms. The school accountability system that Bush set up, the Florida Comprehensive Assessment Test, is scandal-ridden, but he still travels the country promoting his test-based reform model.

Louisiana is the state where Republican Gov. Bobby Jindal instituted a statewide voucher program that gave public money to scores of Christian schools that teach Young Earth Creationism, the belief that the Earth and the universe were created by God no more than 10,000 years ago. Kids learn that dinosaurs co-existed with humans. That’s the state that got Rhee’s top grade.

A quick digression on some good indicators of when a metric has been cooked:

1. It reaches an unexpected, even unbelievable conclusion in the favor of the person designing the metric;

2. It leaves out important variables;

3. And leaves in other variables only tangentially related to the central question;

4. It uses an odd, difficult to justify weighting scheme, making certain factors dominant for no apparent reason.

This report is not only cooked till it's charred; it also flies in the face of Rhee's own rhetoric on tests and accountability. It is, in a word, indefensible, but just as Ryan had James Stewart, Rhee has Matt Yglesias.

Michelle Rhee is a controversial figure, and anything her advocacy organization, Students First, does is going to attract a lot of derision. But having had the chance to play around with their "report card" on state policy, I think there's a lot to like here.

The best thing about it, really, is just that they did it. Importantly it's a report card assessing the state of education policy in different places, not outcomes. ... Only two states score above C+ on their ratings—Louisiana and Florida—and student learning outcomes in those states are far from the best in the nation. If Louisiana starts making a lot of progress in closing the gap with, say, Maryland, then that'll be powerful evidence for the Students First approach. But if it doesn't, then you get the reverse.
...
In policy terms, the most interesting thing about the Students First report is probably its treatment of charter schools. ... The Students First perspective more wisely dings states that make it too hard to open charters but also dings states (like, say, Arizona) that do much too little to hold charter schools accountable for performance.

You should probably read the whole thing (it's less than 300 words) but this gets at the gist. The entire piece is pretty much just a pander and two short, flawed arguments.

Let's start with the "powerful evidence" argument. Yglesias here treats the report card as not really being a measure of school quality (he doesn't have much choice since the score is actually inversely related to school quality), rather a measure of where schools fall on a policy spectrum so we can basically treat their score as an independent variable when evaluating these policies by comparing score with improvement.

It's worth noting that Rhee's site introduces the report card as follows "We hope this helps reveal more about what states are doing to improve the nation’s public education system so that it serves all students well and puts each and every one of them on a path toward success." Here and elsewhere, Rhee clearly means that the states with better scores are doing a better job. This doesn't align very well with Yglesias's argument.

More to the point though, the argument doesn't hold up even in isolation. The idea of providing a useful indicator would only make sense if we scored the schools at the beginning of implementation of the policies. Instead we have a collection of initiatives with varying start dates, most a few years old, some dating back to Jeb Bush. Perhaps as bad, Yglesias leaves the time frame open (always a bad idea) in a situation where a shake-up in the achievement rankings for any reason will tend to favor states at the top of the Students First list. (Louisiana can't really go that far down.) Any kind of causal inference drawn from a change in one of these top scored states would be meaningless.

The only other specific Yglesias can come up with is that the report supposedly requires schools to hold charter schools accountable for performance. Putting aside the obvious "accountable for performance" irony, this claim is a bit difficult to accept at face value given the related question of holding private institutions that receive state money accountable. Remember, Louisiana is Rhee's top ranked state despite a voucher system notorious for its lack of accountability:

The school willing to accept the most voucher students -- 314 -- is New Living Word in Ruston, which has a top-ranked basketball team but no library. Students spend most of the day watching TVs in bare-bones classrooms. Each lesson consists of an instructional DVD that intersperses Biblical verses with subjects such chemistry or composition.

The Upperroom Bible Church Academy in New Orleans, a bunker-like building with no windows or playground, also has plenty of slots open. It seeks to bring in 214 voucher students, worth up to $1.8 million in state funding.

At Eternity Christian Academy in Westlake, pastor-turned-principal Marie Carrier hopes to secure extra space to enroll 135 voucher students, though she now has room for just a few dozen. Her first- through eighth-grade students sit in cubicles for much of the day and move at their own pace through Christian workbooks, such as a beginning science text that explains "what God made" on each of the six days of creation. They are not exposed to the theory of evolution.

"We try to stay away from all those things that might confuse our children," Carrier said.

Other schools approved for state-funded vouchers use social studies texts warning that liberals threaten global prosperity; Bible-based math books that don't cover modern concepts such as set theory; and biology texts built around refuting evolution.

That's it. Out of the "lot to like" here, Yglesias can only come with a flawed we-can-see-how-we're-doing argument and a highly suspect claim about accountability. Less than three hundred words total and he's clearly scraping bottom to put those together.

This whole affair is a case study in how bad ideas lodge themselves in the discourse through journalistic convergence and superficiality, the fetishizing of balance, and the inability of otherwise smart, responsible people to admit (perhaps even to themselves) that they've been proven wrong.

update: link added.