Complex Systems

Libertarian Megan McArdle is in favor of education reform. However, in her latest column she points out the major obstacle faced by reformers:

That's why radical reformers so often end up vilified (and frankly, so often end up making some colossal bloomers; institutions are complicated, and reformer's prescriptions are never as complex as the institution they are trying to change).

I think observations like this elevate the debate. They raise the legitimate question of how do we improve a complex system (my view: incrementally and experimentally). But, as Mark has noted, the debate has often drifted into the surreal with even staunch leftists desperately opposing government regulation of education. It is an odd debate.

XKCD understates its case

There's no reason to resort to a double negative. Just check this out.

At last, something that Paul Krugman and Alex Tabarrok can agree on

Though it involves Middle Earth, not macroeconomics.

One reason to look forward to the Republicans retaking Congress and the White House

At least liberals can be happy again.

A day in the life of a Teach(er) for America -- posted for future comment

I've been meaning to post this for a long time (from Michael Winerip's NYT article on Teach for America):

The 774 new recruits who are training here are housed in Rice University dorms. Many are up past midnight doing lesson plans and by 6:30 a.m. are on a bus to teach summer school to students making up failed classes. It’s a tough lesson for those who’ve come to do battle with the achievement gap.

Lilianna Nguyen, a recent Stanford graduate, dressed formally in high heels, was trying to teach a sixth-grade math class about negative numbers. She’d prepared definitions to be copied down, but the projector was broken.

She’d also created a fun math game, giving every student an index card with a number. They were supposed to silently line themselves up from lowest negative to highest positive, but one boy kept disrupting the class, blurting out, twirling his pen, complaining he wanted to play a fun game, not a math game.

“Why is there talking?” Ms. Nguyen said. “There should be no talking.”

“Do I have to play?” asked the boy.

“Do you want to pass summer school?” Ms. Nguyen answered.

The boy asked if it was O.K. to push people to get them in the right order.

“This is your third warning,” Ms. Nguyen said. “Do not speak out in my class.”

Lots of interesting jumping off points here about Teach for America, classroom life and how not to handle a math lesson. Now if I can just find time to write them up.

"We've made huge advances in what they're called" -- New Republic edition

This post by Joseph got me thinking. Charter schools are private contractors providing services that were previously provided by the government. Any statement that's true about charter schools should still be true if you substitute in the phrase "some private contractors."

But if you actually make the substitution, you often end up with statements the author would never think of making. Statements like this:

But Mead says ... she’s seen Gray hint that he’d like to more tightly regulate [private contractors]. “We have a law that gives a tremendous amount of autonomy to the [private contractors] but enables them to implement programs that can be effective. If you try to put more regulation on that, if can dissuade people from [privatizing],” Mead says.

Would Seyward Darby normally describe a push for tighter regulation of private contractors as "disappointing"? Would the New Republic normally endorse a candidate because he was against stricter regulation of private contractors? Would everyone take a moment and see if Rod Serling is taking a smoke break in the vicinity?

I strongly believe that there is a place for charter schools in our system, but those schools have to meet exactly the same criteria as other contractors. Two of those criteria are transparency and openness to regulation, and given recent history, it's safe to say that some charter schools are failing these tests.

[note: typo in the title has been corrected]

Tenure Track Jobs

Somehow the blogroll for Prof-like Substance isn't updating so I missed this post. But it is excellent advice. The worst thing that can happen is that people decline your applications. But it can be remarkable how somebody can end up as a great fit in a place that they never expected . . .

Implications of the DC primary on education

Mark and I have been talking a lot about education. Today Seyward Darby has an article on the implications of Michelle Rhee leaving. What struck me as most interesting is that there are three concerns raised:

1) The fate of the new teacher evaluation system

2) That performance pay, based on IMPACT score, might not go forward

3) There could be increased regulation of character schools

Mark has commented on issues with a focus on narrow methods of teaching evaluation. But I think it is safe to say that items 1 and 2 depend critically on the quality of the method used.

As for number 3, deregulated private educational entities have geenrated concerns as the quality of instruction is hard to measure.

So everything hinges on how good the IMPACT evaluation score is. Now, this score is a hybrid (as opposed to pure standardized test scores) so it might not be awful:

It rates its teachers based partly on how well they improve student learning from year to year, and partly on intensive classroom observation by their supervisors.

However, it does not address the concern about teachers learning to game the system. Many ranking systems work very well at first and then people learn to optimize against them. Just consider how many articles there are on how to improve your FICO score.

But the core ideas remain to find a way to measure teacher performance on some sort of low dimensional scale and to reduce regulation of private sector competitors. It is unclear that these approaches will get at the key issues, even if (due to an extraordinarily dysfunctional school system) they may produce some improvements in the Washington DC schools.

More on the SRI KIPP report

This is a fairly small case of researchers being (deliberately?) obtuse, but it's representative enough to merit a brief post.

First look at this passage from Wikipedia:

Some observers, such as the authors of The Charter School Dust-Up,^[5] say that KIPP's admission process self-screens for students who are both motivated and compliant, from similarly motivated and compliant—and supportive—families. Parents must commit to a required level of involvement, which rules out badly dysfunctional families.

You'll notice this is consistent with some points Joseph and I have been arguing for awhile about how selection and peer effects can affect a school.

Now pay close attention to the following from SRI:

KIPP schools’ higher-than-expected test score results draw both attention and claims that they “cherry-pick” high-achieving students from poor neighborhoods. This is the first report to closely scrutinize the praise and criticisms associated with KIPP, as well as key challenges facing Bay Area KIPP schools today.

In the three KIPP schools where they were able to draw comparisons, SRI researchers found that students with lower prior achievement on the CST were more likely to choose KIPP than higher-performing students from the same neighborhood, suggesting that, at least at these schools, cherry-picking does not occur.

Did you catch the shift? Like most areas, academic success is largely determined by attitude and work habit. This is particularly true in a school with extended hours of instruction and a longer year. There is every reason to believe that these traits are strongly selected for here, just as suggested by the Economic Policy Institute.

The Charter School Dust-Up came out from the EPI in 2005, three years before the SRI report, but the SRI researchers never directly acknowledge the EPI's conflicting findings. Instead they pull the researchers' version of the old school kid's trick of pretending to mishear an inconvenient question.

Calling Yves Smith

(Via Andrew Gelman) John Sides has good take-down of Steven Hayward's odd screed for the American Enterprise Institute's house journal. Here's a choice exchange.

From Hayward:

Of course, this is just the silly stuff. The real problem with academic political science is its insistence on attempting to emulate the empiricism of economics and other social sciences, such that the multiple regression analysis is considered about the only legitimate tool of the trade. Some regressions surely illuminate, or more often confound, a popular perception of the political world, and it is these findings Klein rightly points out. But, on the other hand, I have often taken a random article from the American Political Science Review, which resembles a mathematical journal on most of its pages, and asked students if they can envision this method providing the mathematical formula that will deliver peace in the Middle East. Even the dullest students usually grasp the point without difficulty.

And from Sides:

This is sort of bizarre. Let’s leave aside the notion that “multiple regression analysis” is the “only legitimate tool.” That’s the impression of someone who doesn’t read much political science. I’m more interested in Middle East peace. Here’s my question: if Hayward picks up the American Economic Review, does he envision that their mathematical formulas will produce global prosperity? That’s the standard to which he seems to hold academic research. If so, he should be disappointed by virtually the entire corpus of social science, and perhaps by a decent bit of the hard sciences as well. After all, there’s still that cancer thing.

I'm going to have to take Sides' side here (and not just for the wordplay), but what really struck me as amusing was the use (starting with the subtitle) of economics as the gold standard of empiricism. Is that really the way you want to play it give the current state of economics in general and macroeconomics in particular?

This is so funny it made me cry

From Dr. Girlfriend.

Is this not every bio-statistician’s worst nightmare?

They just don't care

I am sure that most people in the news media watch every episode of the Daily Show and the Colbert Report. Then they get up the next morning, go to work, and do their God-damnedest to drag American journalism to its lowest point since Benjamin Franklin Bache accused George Washington of collaborating with the British.

They know what they're doing. They just don't care.

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
Islamophobiapalooza
www.thedailyshow.com
Daily Show Full Episodes Political Humor Tea Party

So which one is it?

DeanDad brings this point up in a post today. It's not related to the main point of the post but it is certainly worth noting:

Hell, while we’re at it, let’s make a point of generating enough math teachers so that every state in the country can require four years of math in high school. Get the public K-12 system up to basic competence, and see what happens.

I think, at some level, we need to decide what the priorities in education are. Is there a shortage of competent math teachers? If so then a process in which we make it harder to become a teacher would seem to be counter-productive. After all, we are also grappling with real issues of potential reductions in the number of teachers due to budgetary constraints.

I think that failing to decide on these issues are at the heart of my concerns about education reform. Reformers point to schools like KIPP that have attrtion rates that make them infeasible as a national model. So that doesn't seem to be a way forward. But in the actual education system, we are discussing reducing resources and trying to compensate with higher quality.

Now, add in that the metrics used to evaluate schools have serious concerns (as pointed out by a wide variety of researchers) and it gets hard to see what the road forward looks like. Clearly, if we are reducing resources to education then we can't achieve this with a higher investment. We might gain some efficiency by breaking contracts with current teachers (over tenure and pensions) but such actions tend to increase costs in the long run.

So I think that the real thing that I want to see out of educational reform is specific proposals. Honestly, I suspect that a series of initiatives at the school district level (focusing on the issues in each area) might be the way to go. But I worry that the current approach seems far too focused on test scores and not on the actual process of education.

The appeal of bad ideas -- why do rational executives keep trying ideas that never work?

A while back my friend Brian (of Ultrasonic Remote) and I found ourselves on the subject of TV remakes, shows that kept the title and general premise of an old hit but rebooted it with a new cast. After a while a pattern became obvious, the few successes all had something in common: they were all science fiction or fantasy shows (and even there the odds were not good).

When we went out of that subgenre, we couldn't come up with a single success. Many of the shows were high profile affairs with talented casts (Kojak, Dragnet, the Fugitive) but we are still talking about a failure rate of pretty much 100%.

Eventually, some show will break the streak. Perhaps it will be Hawaii Five-O or the new Rockford Files (after all, how hard could it be to replace James Garner?), but this still begs the question, why do highly paid executives in a tremendously competitive field insist on trying things that have repeatedly and consistently failed?

(Acknowledging) Failure Is Not an Option

Here at OE, we have been spending a lot of time discussing the horrible student attrition rates in Bay area KIPP schools, but when I first came across the statistic, I hesitated before using it in a post. The number was so absolutely damning that I had to wonder about the source. Had some anti-charter school group gone through all the data and cherry-picked the worst thing they could find?

So I looked up the source of the statistic and found a veritable sales brochure for KIPP. The article opened with the headline:

"New Study Finds San Francisco Bay Area KIPP Students Outperform Peers"

followed by the subtitle:

"Combination of Key Features Contributes to Success, Provides Lessons for Other Public Schools"

The rest of the article continues in the same vein before finally getting to this:

As researchers analyzed the student achievement data and KIPP’s approach, they also identified challenges facing Bay Area KIPP schools, including high student attrition rates, teacher turnover, and low state and local funding. For example, 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade. Annual teacher turnover rates have ranged from 18 to 49 percent since 2003-04.

Putting aside the human cost paid by the students who fell into that sixty percent and their families (which is huge), the selection/attrition process described in this post results in a school filled with roughly the top quartile of hard-working, dedicated students. All but the most incompetent administrators will have spectacular results under those circumstances. The performance described here ("In most grades, Bay Area KIPP students make above-average progress compared with national norms, and four out of five KIPP schools outperform their host district.") is, if anything, on the low end of what we would expect yet it is written up in the most glowing terms imaginable.

When it comes to charter schools, there's a long tradition of bad news stories with positive or at least neutral headlines. Consider this article from the New York Times. The key paragraph was:

But for all their support and cultural cachet, the majority of the 5,000 or so charter schools nationwide appear to be no better, and in many cases worse, than local public schools when measured by achievement on standardized tests, according to experts citing years of research. Last year one of the most comprehensive studies, by researchers from Stanford University, found that fewer than one-fifth of charter schools nationally offered a better education than comparable local schools, almost half offered an equivalent education and more than a third, 37 percent, were “significantly worse.”

And what was the headline for this grim finding?

"Despite Push, Success at Charter Schools Is Mixed"

I genuinely believe that there are some wonderful charter schools out there and we can greatly improve education by building on these successes, but in order to do that we need to recognize and honestly admit the failures.

Up until now, that honesty has been in short supply.