Thoughts on the NYT's attempt to reboot the Airport franchise

I first heard about this through Brad DeLong who did, I will admit, have me going until I clicked through the link to this James Fallows piece. Fallows, following the lead of Gene Weingarten and Jim Romenesko*, had some questions about this dramatic account entitled "The Plane Was About to Crash. Now What?" by Noah Gallagher Shannon.

Actually, 'dramatic' is  not really an apt description. A better word would be 'melodramatic/' complete with details like

The captain came out of the cockpit and stood in the aisle. His cap dangled in one hand. “All electricity will remain off,” he said. Something about an open current and preventing a cabin fire. Confused noises spread through the cabin, but no one said a word. “I’ll yell the rest of my commands from the cockpit.” I could see sweat stains under his arms. “Not going to sugarcoat it,” he said. “We’re just going to try to land it.”

(it later came out that the plane was an Airbus 320 which is a large enough plane to raise questions about the shouting-over-your-shoulder means of communication.)

Lots of these detail raised questions, particularly with real aviation experts like Patrick Smith, who demolished the NYT piece until there was little left than a pile of phonemes, then followed up by talking to someone who actually had access to the maintenance record:

According to the information I was given, the pilots' post-flight logbook entry, which references a caution message displayed on a cockpit advisory screen of the Airbus A320, reads as follows:

"ECAM HYD Y RSVR LO LVL"

What this means, essentially, is that one of the plane's three main hydraulic systems was indicating a low level in its fluid reservoir. Airbus color-codes its hydraulic systems; this would have been the yellow (Y) system.

Per checklist instructions, the crew would turned off this system off. This is unusual, but the loss of a single hydraulic system on a modern airliner is far from a serious emergency. All critical components have at least one alternate source of hydraulic power.

Further, the corrective action note in the logbook implies the issue was merely an indication problem. Fluid quantity was found to have been at the normal level all along.

But perhaps most importantly, Shannon's essay revolves around a supposed landing gear problem. As I explained earlier (below), even the most serious landing gear malfunction sits pretty far down in the hierarchy of potential disasters. Even lower, however, is a landing gear problem that does not exist: the yellow hydraulic system does not power the landing gear.

An A320 captain I spoke to says that a shut-down of the yellow system would have meant, at worst, a slightly longer-than-normal landing roll (due to loss of the right engine thrust reverser and some of the wing spoiler panels), and, in newer A320s, loss of the nose-gear steering system, requiring a tow to the gate.

Shannon had thrown up enough red flags to begin with, but this puts it over the top, tilting his entire account from one of eye-rolling embellishment toward one of outright fabrication.

What really caught my eye, though, was the NYT editor Hugo Lindgren's response to Fallows:

Some commenters have seized on certain details of "The Plane Was About to Crash. Now What?" by Noah Gallagher Shannon in order to question whether this emergency landing happened (and perhaps even whether the author was on the flight). But there is simply no question. The author was on Frontier Airlines flight #727 on June 30, 2011, from Washington to Denver. It was an Airbus 320. The author sat in seat 12A. This flight was diverted to Philadelphia. The FAA reports that the pilot declared an emergency due to a low hydraulics indicator light and that upon landing the plane needed to be towed to the gate. Frontier airlines confirms that an Airbus A320 experienced "a maintenance issue on departure from Washington DCA. The flight diverted to Philadelphia due to easier access. The aircraft and all passengers landed safely."

Did the author's personal recollection represent an accurate picture of what he experienced on that flight? Well, only he can attest to his own experience. But the author did provide receipts and took notes after the flight to back up his account. And his recollection, when run by an aviation specialist, did seem entirely plausible to him. While some of the author's language may have been imprecise, his recollection of his experience was consistent with recollections of passengers in similar air incidents. Naturally, not every detail matches everybody else's experience. Surely even people on that plane would remember it differently. The story was about the personal experience of a fearful moment. The author did not present himself as an authority in airline technology or emergency procedures. The airline, in fact, refused his request for more information about what happened after the fact. He only reported what he heard and felt, which is consistent with the magazine's Lives page, where the account was published.

Of course, out of the tons of criticism from Fallows, Smith, Weingarten, Romenesko and company, the suggestion that Shannon had made up the flight was, at most, a tiny and generally implicit part of the case against the article. Lindgren is using some rhetorical misdirection here; he's just not doing it very well.

More troubling is the now familiar nonchalance of journalists toward accuracy. "Perceptions are subjective." "Opinions differ." "Who are we to judge whose claims are valid?"

But of they do judge. Only a select few articles make it into the New York Times. By publishing those articles, the people who run the paper are putting their reputations behind those pieces. If they want to maintain those reputations, those people need to be more careful about what they publish and less weaselly when something like this slips by.

* Make sure to check out Jeremy Repanich's comment.

Property Rights

There is an argument that has been going around that is, roughly speaking, "if Libertarianism is so great why is no country organized along these lines".  Now, it is worth noting that this comment is only true with respect to extreme or Utopian forms of Libertarianism; the movement itself has done a lot to shape a number of modern societies.  But it is true, that on first glance, this method of governance seems scarce on the ground.  Now this isn't a fatal objection, outside of Athens there were few obvious Democracies before the High Middle Ages (and the flaws of the Athenian form of Democracy are only imperfectly present in modern representative democracy). 

Mike Konzal makes the claim that Feudalism is actually a good example of Libertarianism.  Part of me wants to dismiss this claim as "too cute by half".  I also note that it immediately offends a lot of Libertarians.  And I think that is a shame as I suspect we can learn a lot from the actual analysis.

The discovery that Democracy could end up as a form of impulsive mob rule comes directly from the Athenian adventure.  Modern democrats did not reject the system because of this (potentially terrifying) flaw but rather used it to understand how to build a better system. 

So consider this argument from Konzal:

When libertarians say they are for basic rights, what they are really saying is that they are for treating what liberals consider basic rights as property rights. Basic rights receive no more, or less, protection than other property rights. You can easily give them up or bargain them away, and thus alienate yourself from them. (Meanwhile, all property rights are entirely fundamental - they can never be regulated.)

Now there may be Libertarians for which this is a feature and not a bug.  But just like modern forms of Democracy had to grapple with the issues of mob rule as seen in Athens (the death of Socrates, the insane idea of invading Sicily), understanding how this kind of system might work can actually bring out some of the key features that need to be thought through.

Two things come immediately to mind.  Property rights in the feudal era got enormously complicated, with specific places coming with extremely complex ownership rules.  People could own the right to do one thing in a certain place but not own the underlying land.  Feudal relationships were confusing, to say the least.  I think this needs to be considered.

The other issue is whether or not there should be rights in balance with property rights.  The idea that property rights are fundamental does not mean that they should be the only item in consideration.  Democracy sacrificed the purity of open voting on all matters for additional stability.  I think that these issues could be thoughtfully addressed by Libertarians looking to build a feasible model.

I do think the Randian/Objectivist ideal doesn't exist for a reason.  I can also see reasonable people not liking the core idea of property rights.  After all, unequal starting points can lead to very unequal outcomes.  But I think that this analysis should be a challenge to understanding and enhancing the operationalization of the Libertarian model.

More adventures in intellectual property --- Ghost Rider rides again

Not long ago, Marvel managed to get a couple of unlikely hit movies out of a fairly obscure motor-cycle riding character called. Ghost Rider. That unexpected success has led to some lawsuits over who owns the character.

NEW YORK (Reuters) - A comic book writer who claims he created the flaming-skulled character called Ghost Rider got another chance to press his claim on Tuesday when a U.S. appeals court revived his lawsuit against Marvel Comics.

The 2nd U.S. Circuit Court of Appeals vacated a 2011 district court ruling that found the rights to the character belonged to Marvel Comics, owned by The Walt Disney Co. It returned the case to the lower court for trial.

Former Marvel freelancer Gary Friedrich, who claims he created the character, will pursue the case "aggressively and vigorously," his lawyer Charles Kramer said.

Ghost Rider, a motorcycle-riding vigilante whose head is a flaming skull, first appeared as a vigilante superhero in 1972.

Sorta.

It turns out that the 1972 character was something of a reboot of a Western-themed Ghost Rider Marvel launched in 1967 (again with the involvement of Gary Friedrich) and that's where the intellectual property question gets amusing because that incarnation was perhaps the most blatant example of plagiarism in what has always been a plagiarism-fueled medium.

Stan Lee's claims in the Sixties that he didn't read other comic books were a lot like Milton Berle's claims that he didn't listen to other comics' material. Lee had spent two decades in a closely-knit industry jumping on trends and borrowing ideas, starting with DC and EC derived books and ending with an unacknowledged appropriation of Julius Schwartz' successful formula for rebooting Golden Age heroes with science fiction origins and streamlined looks.

The difference between Berle and Lee was that a large part of Lee's audience didn't get the joke. Many of them had gotten serious about comics in the mid-Sixties when Marvel had hit its stride and really was putting out the best and most innovative comics. For them, Lee's comically over-the-top claims seemed entirely reasonable.

It's safe to assume that in 1967, few of those teen aged fans had any idea that Marvel's new Western themed hero, Ghost Rider, had a strong resemblance to another company's hero, a resemblance that included having the same name, costume, concept, atmospheric look, and artist. The retread only ran for seven issues but that was enough for Marvel to claim ownership of the name and, a few years later, launch another Ghost Rider.

(Yes, that is a Frazetta cover)

And here's the Marvel version:

Of course, the idea to introduce a character by this name in 1949 may have had something to do with a hit song introduced in 1948 (and covered a billion times since).

When the guys who are in for murder one start arguing about Jane Austen...

This is a bit outside of our normal education beat, but this account of a man's experience teaching English and Philosophy to lifers in a British prison is definitely worth checking out.

From American Public Media's the Story.

Evolution and the breakfast landscape

Here's a question about evolutionary psychology. Not a rhetorical question or a snarky question (even though I have been a bit snarky on the subject in the past). Everyone knows why we evolved to seek out salt, sugar and fat in our diet, but why did we evolve to favor an extremely rugged yet malleable culinary landscape?

Take breakfast, for example. Sometimes (though not often) I'll hit a diner and say what-the-hell and get coffee, juice, eggs, bacon, hash browns, and biscuits. With the exception of the juice and the bacon (which are already optimized), I will add the right level of flavorings to each dish to get it to its landscape maxima (sugar for the coffee, salt and pepper for the eggs, hot sauce and ketchup for the hash browns, jam or maybe even sausage gravy for the biscuits).

The optimization is done on a dish-by-dish basis and is, for the most part, independent. If only eggs are available, I'll have a sugar-free breakfast. If all I have around the house is coffee and fruit, I'll have a low-sodium breakfast. Though there's an evolutionary imperative that makes me crave sugar and salt, it is somehow dependent on the other coordinates of the landscape. I want sugar but not salt with my coffee, salt but not sugar with my eggs and both salt and (caramelized) sugar in my hot chocolate.

Small children complain loudly and adults mumble grumpily when certain foods mix on a plate even though, as parents often remind us, it all goes to the same place.

How did our places get so context-sensitive?

Matt Yglesias may not be helping his cause

There was an odd exchange recently between Diane Ravitch and Matthew Yglesias.

Ravitch wrote a post about James Cersonsky's American Prospect article on Teach for America's political power. She introduced the post with this:

Teach for America began with a worthy goal: to supply bright, idealistic college graduates to serve in poor children in urban and rural districts.

But then it evolved into something with grand ambitions: to groom the leaders who would one day control American education.

Yglesias's response is rather strange. He doesn't mention Cersonsky or the American Prospect his post but  he only explicitly addresses points that come from Cersonsky's article; not from Ravitch. I say explicitly because the example Yglesias uses is certainly relevant to Ravitch's claim (though definitely not in the way he intended).

I thought of this over the weekend at my college reunion, where I met up with an old friend of mine who right after graduation was a science teacher in a public school in New Orleans. Later, she taught at a KIPP-affiliated school turnaround venture in New Orleans and then became founding assistant principal of a KIPP-affiliated school there. Then she moved back to the Boston area and became principal of a charter school called Excel Academy. Now she's a fellow at an nonprofit called Unlocking Potential, but soon she's going to become principal of a troubled public middle school in a a Massachusetts town whose school district has been placed in state receivership.

The part about reunion caught my eye. That's a lot of jobs for a 2003 graduate (Matt Yglesias '03 as they say at Harvard) so I did a little digging. I may have missed some important details but here's what turned up:

Barring a really astounding coincidence, Yglesias is talking about an educator named Komal Bhasin. Here's Bhasin's job history:

She taught from 2003 to 2005.

With a bachelor's degree and two years teaching experience, she was named assistant principal of a school.

With a bachelor's degree, two years teaching experience, and two years experience as an assistant principal, she was named principal of a different school halfway across the country. (You will often find sudden promotions within a school where you're dealing with known quantities. Putting a fledgling assistant principal in charge of a different school in a different region is much more unusual, particularly an administrator with almost no teaching experience.)

With these qualifications, and five years experience as a principal, she got a principal-in-residence with a high-profile education reform institute -- a relatively short tenure and thin resume for this kind of position.

Obviously, there's a limit to how much we should infer here, but Bhasin has indisputably gotten a series of promotions that were surprising given her job history, education and (as far as I can tell) publications and she has also gotten considerable exposure as a rising star in the reform movement .

Just to be clear, I am sure that Komal Bhasin is a smart and dedicated educator and may well be an excellent administrator. Nothing should take away from that, but it is also true that, given what we know, her career path would seem overwhelmingly to support the idea that she was being groomed for a Michelle Rhee type leadership role just the way Ravitch suggests.

In other words, Yglesias came up with a great example, just not for his side.

Investment advice from Alfred Hitchcock

Mail Order Prophet, a 1957 episode of Alfred Hitchcock Presents written by Robert C. Dennis (teleplay) and Antony Ferry (story) and starring E.G. Marshall and Jack Klugman

Appeared previously in You Do the Math.

"How do you make money doing this?" The answer is simple...

If they hadn't used cast members, I wonder how many viewers would have caught on?

Thoughts on Ray Fisman's much alluded-to charter school article

As mentioned before, Ray Fisman has a piece up at Slate discussing charter schools and it's quite good. Arguments for both sides are spelled out in a clear and balanced manner and, with a few exceptions (and only one of those major), data issues are addressed as thoroughly as is possible in a general interest publication.

That major issue is cheating and data manipulation and it's the one problem I would most like to get Fisman's take on, partly because I really do believe it's probably the biggest challenge facing the reform movement but mainly because of Fisman's expertise in the question of how best to set up metrics and incentives in order to keep things running smoothly.

My other issues are more the sort of technical points you probably couldn't get away with in Slate. For example, take a look at the following from Fisman:

At least part of the disagreement revolves around whether charter schools deliver on their promise to improve student outcomes. You might think this is a relatively easy proposition to evaluate—just compare whether charter school kids do better on tests than those in public schools. But any effort to compare performance is confounded by the fact that the kinds of parents who take steps to enroll their children in charter schools may be the kind of motivated and supportive parents whose children would have done just fine in any school system. (In the current study, charter school applicants do in fact have higher than average test scores even before they enroll. However, other analyses have seen charter school applicants with below-average scores, perhaps because kids struggling in the public system are more apt to look for other options.) And if the longer hours and additional school days that are a feature of many charter schools lead underperformers to drop out, the select group that remains may again be made up of those who would have tested well in any school environment.

This isn't just a question of underperformers. Based on my classroom and tutoring experience, I strongly expect we're seeing a strong interaction between treatment and selection. Different kids respond differently to different approaches. There's a very good chance that those kids (and families) who volunteer for more time and more work are also the kids who respond best to that approach. This is not an argument against charter schools; it is, however, an argument for caution in generalizing results.

A somewhat more serious issue shows up in the next paragraph

But for at least a subset of charter schools, researchers can come fairly close to running a clinical trial where some applicants are enrolled at charters and others are left in the public system purely by chance. The reason is that many charter schools are oversubscribed, and their scarce spots are allocated through a lottery. So whether a particular student gets assigned a slot at the charter school is luck of the draw. (This randomness in gaining admission to sought-after charter schools has been documented recently in the films Waiting for Superman and The Lottery.)

Numerous studies have used this lottery method to analyze the impact of charter schools on standardized test scores, and by and large they report similar findings: Charters in rural or suburban areas don’t do any better than public schools, while in urban areas they are associated with greater test score improvements in math and language. But another important point from past studies is that there is enormous variation in the effectiveness of charter schools. There are some great ones but also some real duds.

Normally, in clinical trials, we can treat our subjects as independent. After all, my odds of developing a side effect should have nothing to do with whether or not you developed that side effect. There is simply no way you can assume that the behavior of one student is independent of the rest of the class and that lack of independence can be extraordinarily difficult to account for. While useful, the lottery method is a long way from giving us the results we would expect from a good clinical trial.

But, that aside, Fisman's point about urban schools is tremendously important and is explored in more detail later:

Some charter school advocates will surely point to the new study as yet more evidence that public school districts should be replaced by a more decentralized approach to education, with a greater emphasis on charter schools. But that’s a very narrow interpretation of the mounting evidence of charter schools’ successes. Focusing on these successes glosses over the many cases where charter schools fail to outperform their public peers. For suburban districts in Massachusetts, for example, the numbers don’t favor the charter school advocates, in large part because suburban public schools in the state are pretty effective already.

This is a huge point and it addresses one of the biggest concerns of reform skeptics, namely that real problems might be used as cover to change the parts of the system that are working but which are ideologically disagreeable to some of the reformers and, as a result, leave a system more broken than it is now.

From the beginning of the debate, the question of how to close the achievement gap has gotten conflated with the question of general education reform. The wretched state of education in high poverty areas was often used to argue that we needed to make immediate sweeping changes to all areas of education and some of those changes were truly radical.

Until recently, many of the reform proposals (firing 80% of new teachers, extensive privatization, effectively removing teachers' right to unionize, and quite a few more) allowed for little if any middle ground, particularly given the amount of data that was not breaking in the reformers' favor. I strongly suspect that Diane Ravitch's Road-to-Damascus conversion came from her inability to move gradually away from her previous positions.

I realize I'm reading a lot into one recent article, but this seems to be where serious reformers like Fisman and Jonah Rockoff are now:

1. Charter schools have a great deal of promise;

but

2. They aren't right for all kids or all situations;

3. and there may be scaling limitations both in terms of teachers and parents;

4. so we should focus on areas where the need is greatest and the evidence for effectiveness is strongest.

This is not the exact list I would have made (OK, technically I did make this list but you know what I mean), still there's nothing here that I would disagree with. What we have now are questions of degree, which tend to be more productive and are much more likely to be convergent.

Which would be nice for a change.

Give my regards to bean town, but...

I found quite a bit to agree with in Ray Fisman's recent piece on charter schools (and even more to like about it), but I do have a couple of issues I wanted to get out of the way. The first involved the threat to reform presented by people who cheat and/or game the system. The second revolved around some of the issues with the heavy reliance on a small set of Boston schools as examples.

I haven't had time to go through the Boston report in detail, but assuming that it's solid (and I have no reason to think otherwise), there are still a couple of reasons to be careful about drawing overly broad inferences from this particular city at this particular point in history.

The first (which is pretty much self-evident) has to do with Boston's unique relationship with higher education. The second, and more important, goes to the heart of the problems with modern urban schools.

Here's Fisman:

Why wouldn’t parents try to get their children into charters that would get the kids on track to go to college? One explanation is that the debate about whether charter schools “work,” with its focus on testing and college placement, loses sight of the many reasons why people choose a school and what they value in an education. Charter schools can be set up for any purpose. Some focus on the arts, others emphasize cultural heritage (there are multiple Hmong charter schools in the Twin Cities alone); some are vocational, others rigorously academic. As Rockoff puts it, asking whether a particular school is good based on test scores or college placement is in many cases the wrong question: “To extend the restaurant metaphor, some people like Italian, others like Thai food. Similarly, many [charter] schools focus on tested subjects, while others might emphasize creative writing or the arts.” Not every charter school is right for every kid. Nor would it be an unmitigated good if all parents could segregate their children by their interests and beliefs.

But a big reason, maybe the big reason, that urban parents send their children to charter/magnet and parochial schools is left out of that paragraph. In general, parents want their children to be successful happy and safe, but of those, safety is the most immediate. These non-standard schools usually have a low or no-tolerance approach to gangs. More significantly, there's a huge anti-gang selection effect.

If you want to compare different urban education initiatives, you have to take into account gang activity and neighborhood violence, not just in terms of absolute levels but in terms of attitudes and expectations, and, while I have no expertise in the area, I do know enough to know that understanding urban crime in Boston over the past twenty years, a period that saw both the advent and apparent collapse of the "Boston Miracle," is something that does require considerable expertise.

None of this undercuts the findings of the study, but it does greatly complicate the process of drawing inferences from it. It's possible that what worked in Boston won't work as well in most urban areas. It's also possible that what worked there will work even better elsewhere. I'm not being cute here -- I could honestly see this going either way -- I'm just trying to emphasis just how messy this data is.

We have here everything your stat professor warned you about. Interactivity. Non-linearity. Heteroskedasticity. Nesting. Selection effects. Missing not at random. Falsification. Hawthorne effects. Volunteer effects. Peer effects. Researcher bias. Sample size issues.

Though I'm not a Bayesian by training, I think the appropriate response is at least in spirit, Bayesian. We will have to factor our doubts about the research in when we weigh our conclusions but that's not the same as ignoring those conclusions entirely.

This brings me back to what I liked about Fisman's article. It's balanced, it acknowledges the limitations of the data, and it suggests focus on areas where reforms will do the most good if their assumptions are sound and will do the least damage if they aren't.

But more on that later.

Two things you should read in their entirety

The first is Mike Konczal's analysis of the drivers of mass incarceration, "Against Law, For Order"

As Bernard Harcourt examines in The Illusion of Order, broken windows policing is predicated on separating neighborhoods into regular, ordered insiders and disordered strangers. [James Q.] Wilson’s view is that regular insiders are the “decent folks” who need to be protected from the disorder generated by strangers. The police, rather than upholding laws and the rights of citizens, uphold order by regulating the behaviors of disorderly insiders and excluding the disorderly outsiders. Criminals lose their insider status in this telling, and excluding them from the community becomes a goal of law. The approach is based on a privileging of order over law, for a lack of order is what attracts criminal behavior, always waiting in the wings to descend.

The second is this short piece from TPM.

A Texas man, Ezekiel Gilbert, has been acquitted of murder after shooting and killing a 23 year old escort he found on craigslist because she would not have sex with him even after paying her $150 fee.

Gilbert was acquitted under a Texas law that allows you to use deadly force to protect your property during a nighttime theft. In this case, the property was either the $150 or the use-rights to the body of 23 year old Lenora Ivie Frago. Frago said she couldn’t give back the money because she had to give it to the driver who was waiting for her.

At the risk of putting too fine a point on this, these deadly force laws have always been about keeping people in their place, socially, economically, racially. Crimes against people who rank low on the hierarchy are seen as less heinous in this Wilson-like view and Ms. Frago ranked low in the eyes of the jury. I think it's safe to say that if a prostitute had killed a man in Texas because he had refused to pay after sex, she would certainly be in prison and probably be on death row.

Contests big and small

This is a follow up of sorts to the recent Project Gutenberg Project post, not in that it involves similar data or statistical techniques but in that both are part of a loose thread on the possible (and in some cases, inevitable) tranformative movements in statistics and research that are generating a lot of talk but perhaps not enough discussion.

Last time I had a post on how we might want to open big data. This time the focus is big contests. Things like the X-Prizes or proposals to offer awards for the development of new drugs. The qualifier 'big' is important; scale plays a significant role in the second half of the post and I'll get to some smaller contests then. For now though, let's consider big contests for huge organizations.

If you put aside reputational capital (which might be a big deal to a small start-up but which wouldn't have much impact on a big multinational), in order for contests to make sense, you're left with the following two inequalities:

Existing incentives < Cost of development

but

Existing incentives + (Prize * Probability of winning) > Cost of development + Uncertainty penalty

That's the picture for the competitors and it's hard to imagine them being satisfied that often.

What about the party offering the prize? For them, much of the benefit comes from the number of groups competing for the prize; presumably the quality of the winning entry should increase monotonically as you add competitors. Number of entries is, of course, inversely related to cost of development.

These contests have long had a special appeal for freshwater economists, but when you get past the open enrollment aspect, the process looks awfully bureaucratic. The competitions are driven by artificial criteria determined by a committee. What's more the criteria generally only have to be met once during a relatively short time frame rather than over the lifetime of the product. This produces a great incentive to cheat.

If you were to sit down and, given these factors, try to come up with the worst possible product to develop with a contest it would probably be in pharmaceuticals. Drug development has huge costs, is largely limited to a small pool of players and is highly vulnerable to fraud (fraud which comes with substantial social costs). Despite this drugs are one of the most frequently proposed applications of the contest model.

The various X-prizes are fortunately not vulnerable to fraud, but they are also poor choices for the contest model: massive projects (though, with the exception of the tricorder, not necessarily that ambitious from a technological standpoint) requiring huge start-up costs that can be undertaken by only a handful of competitors (think dozens instead of hundreds).

The contest model can, however, make a great deal of sense assuming you can meet the following conditions:

A relatively large pool of potential competitors;

Relatively low cost of development;

A relatively high secondary reward/cost of development ratio (secondary rewards include reputation, access to future contracts, publications and for students, grades and thesis material);

Largely ungameable criteria;

Significant positive externalities (for example, a text mining competition might increase the competitors' programming skills and thus increase the skilled labor pool).

For example, DARPA's Grand Challenge contests met all of these conditions. Hundreds of teams competed in the various events and almost certainly produced more quality research and innovative thinking than we would have gotten spending the same money in a straight grant arrangement.

Here, however we run into a different question of scale, not in participants but in number of projects. The Grand Challenge worked not only because it met the conditions listed above but because it took advantage of a certain slack in the system. There were enough people in engineering and computer science departments who were looking for an interesting, high-profile project and who didn't care that the expected value of the compensation didn't come close to paying for the time, facilities and equipment required to compete. How many of these competitions can we introduce before we start cannibalizing the talent pool?

When done badly, contest-based research makes little sense, but even when done well, it can't really be more than a small part of our overall research strategy.

More thoughts on dumping

I'll have more (mostly positive) on this Slate article by Ray Fisman, but first here's (hopefully) one last point on looting.

As part of a good discussion of the state of charter schools, Fisman uses a not-so-good example:

The secret of many charter schools’ success isn’t a mystery: longer hours and additional school days, which are part of a “no excuses” philosophy that emphasizes frequent testing and often requires even longer days from charter school teachers. When public schools integrate these elements—as in a pilot project run by Harvard economist Roland Fryer in Houston, early evidence suggests those schools are seeing the same gains as high-performing charters.

But the success of charters doesn’t necessarily provide us any answers about what will improve education on a larger scale. KIPP, the nationwide network of charter schools and one of the great success stories of the charter movement, has 125 schools serving 41,000 students, making it less than one-twentieth the size of the New York City Department of Education. It isn’t clear that there are enough teachers willing to put in 10-hour days or enough parents who will force their kids to endure the extended hours of instruction. To this point, one recent study by MIT graduate student Christopher Walters argues that lack of demand by parents will be a significant limiting factor in the further expansion of charter schools.

Before we jump to success story in this case, we need some context. We know that KIPP at the very least tolerates principals who engage in large scale dumping (see here and here) and for reasons I've mentioned before and will recap shortly, dumping is one of the nastiest but most effective ways of gaming the system.

Of course this isn't just a KIPP issue. Here's an excerpt from a letter sent by a principal of an LA public middle school principal to Diane Ravitch:

Since school began, we enrolled 159 new students (grades 7 and 8). Of the 159 new students, 147 of them are far below basic (FBB)!!! Of the 147 students who are FBB, 142 are from charter schools. It is ridiculous that they can pick and choose kids and pretend that they are raising scores when, in fact, they are purging nonperforming students at an alarming rate—that is how they are raising their scores, not by improving the performance of students. Such a large number of FBB students will handicap the growth that the Audubon staff initiated this year, and further, will negatively impact the school's overall scores as we continue to receive a recurring tide of low-performing students.

The specific problem here is that even though principals at most charter schools (including KIPP) are almost certainly concerned about their kids, the incentives are tragically misaligned, encouraging schools to shuffle students in ways that are far worse than zero sum.

If you've taught in one of these achievement-gap-closing schools, you know that a small number of students demand a disproportionate amount of your resources, both in terms of instruction and classroom management (particularly for inexperienced teachers). These same students also tend adversely affect morale and social norms.

While it's true that sometimes a transfer turns out to be a good thing for a kid in the long run, at the time, being forced out of one school and shoved into a strange one is disruptive, stressful and insulting. Kids who were confused by the material will generally find themselves even more confused. Kids who were angry will generally find themselves even angrier. Kids who were overwhelmed and stressed out will... You get the drift.

This is a serious, perhaps even fatal flaw in the current configuration of most reform proposals, a misalignment of incentives that actually allows administrators to improve their schools' ratings and make life easier for themselves and their faculties by increasing one of the worst possible outcomes for students.

It is also a surprisingly easy flaw to fix. You simply assign an appropriate penalty for attrition. The difficult part is getting the issue into the debate.

I should add that most administrators don't treat this question lightly. When I taught at a parochial prep school in Watts, the principal agonized over every decision to transfer out a student and I believe she was much more representative of administrators than is KIPP's Randy Dowell. Most educators will try to do the right thing despite the incentives, but that's not a set-up you want to rely on. Incentives have a way of eroding good intentions, particularly in systems where the ethical players actually have less job security.

At the risk of pounding this point in too hard, you cannot have a reform movement without trustworthy data. The level of gaming and outright cheating we've seen greatly undercuts our ability to make rational, data-driven policy decisions and that level is only going to get worse until we fix the incentive system that's driving it up.

You can listen to this before I do... if you hurry

A couple of years ago, This American Life did one of the best pieces of reporting to date on patent trolling. Now they have a followup which Felix Salmon calls "fantastic," though I'm not sure if he means 'great' or 'the stuff of fantasy' (both would seem to apply).

I just downloaded the episode and I didn't pay these guys a dime:

Host Ira Glass and Zoe Chace from NPR’s Planet Money talk with Jim Logan and Richard Baker of Personal Audio, which claims it holds a patent used by all podcasters. Podcasters, they say, owe them money. Only catch: their company never made a digital podcast or invented a way to download it into a listening device. What they did, back in 1996: they patented the idea that such a thing can be done. Now they’ve asked podcasters to pay them licensing fees.

Thomas Friedman demonstrates the Roommate Effect

I have mixed feelings about criticizing Thomas Friedman. For one thing, it's been done. For another, he did some really impressive reporting on the Middle East and I suspect that, if he stuck to that topic, he would still be adding a great deal to the conversation.

In the role of public intellectual, though, he's pretty much been a disaster (insert Peter Principle digression here), and he keeps coming up with passages that are simply too representative not to use as examples of bad punditry.

Which brings us to the roommate effect. The roommate effect is one of the reasons that people who go to elite schools to tend do well professionally.

Imagine a small town populated predominately by people in their early 20s with similar backgrounds who are new to the area. Young people are good at making friends and this scenario is almost perfect for forming new relationships. You have roommates and friends and friends of roommates and roommates of friends. You meet people in the cafeteria and in the coffee houses and in the bars. You find people with common interests in music or movies or art or sports. These people tend to form much of the base of a social network that you will rely on for the rest of your life.

This part of the experience is common to anyone who has gone to a traditional college. But in an elite school, there is a fairly good chance that a new friends will be someone who is or is connected to someone who is rich/famous/powerful. Playing in a college pool league with the son of a Fortune 500 CEO is likely to be a good career move.

And that brings us to this recent Friedman column (mercilessly but not inaccurately satirized by Timothy Burke). The column is basically an unpaid advertorial for the job placement firm HireArt. The weaknesses of the column are a subject for another post; Friedman's lack of understanding of education and the job market is genuinely profound. However he does manage, quite unintentionally, to make an important point about the way things actually work (emphasis added for those who like to skim):

One of the best ways to understand the changing labor market is to talk to the co-founders of HireArt (www.hireart.com): Eleonora Sharef, 27, a veteran of McKinsey; and Nick Sedlet, 28, a math whiz who left Goldman Sachs. Their start-up was designed to bridge the divide between job-seekers and job-creators.
...

The way HireArt works, explained Sharef (who was my daughter’s college roommate), is that clients — from big companies, like Cisco, Safeway and Airbnb, to small family firms — come with a job description and then HireArt designs online written and video tests relevant for that job. Then HireArt culls through the results and offers up the most promising applicants to the company, which chooses among them.

In case you're wondering, Eleonora Sharef got her bachelor's from Yale.