Anatomy of a spiral

While we're on the subject of costs spiraling out of control, here's a relevant post from Ken Levine (who knows a bit about television and movie production):

I once wrote an independent feature set in Bakersfield. I hired a line producer to come up with a budget. I almost passed out when I saw the final number. $10 million dollars. I was hoping for something like $40 thousand.

So I went through it line-by-line and saw that he approached this as if it were AVATAR. There were thousands allotted for plane flights… between Los Angeles and Bakersfield. First class yet. It’s an eleven-minute flight! Thousands were set aside for gifts. Towncars on stand-by, separate hair, make-up, and wardrobe people for each star.

And this was my favorite: There’s a half-page scene where a character comes out of a club at night following someone and discovers it’s so foggy he can’t see his hand in front of his face, and of course he loses the person. (Thick Tulie Fog is a Central California staple in the spring.) Again, a half page scene. The producer had it budgeted for $1 million. This was the conversation (almost verbatim):

Me: Why?

Producer: Are you kidding? Do you know the amount of fog machines I would have to rent to make fog that thick in an open area… and sustain it? Not to mention renting them from LA and hauling them up here and hiring extra personnel to man them. This is a huge undertaking. I hope I can do it for just a million.

Me: Uh huh. Okay, fine. But let me ask you, is there possibly any other way? Can you think of any other options for doing this scene?

Producer: No. Not really.

Me: (exploding) It’s FOG! We can’t SEE anything! Shoot it in the corner of a sound stage with one fog machine! Do it optically and don’t film anything! It’s FOG. At NIGHT.

Needless to say, I did not use his budget.

Required Reading

Alex Tabarrok. 

For the record, I partially disagree with this approach as I think we can go a long way under the assumption of homogeneous response and looking at average causal effects of medication treatment.  In a lot of ways this simplifies what could be a confusing nightmare to regulate.  It's a classic balance of harms: omitting some medications (false negatives) to reduce the number of drugs that are actually harmful (false positives).  Without infinite power you need to draw a line somewhere.

But if you are a defender of the traditional FDA it is actually imperative that you become familiar with the argument from personalized medication.  After all, it is the approach that is going to find all of your errors and may well lead to the updating of the current consensus. 

The looting phase of (higher) education reform -- the opposite of cost disease

The tuition spiral is a horrendously complex story involving organizational theory and economics and politics and culture, but there are some fairly simple conclusions we can draw with a fair amount of certainty. One of those is that the standard cost disease explanation -- that a lack of technology-driven productivity gains are driving the increase -- fails to meet the facts in at least two respects: first, the compensation of those who do nothing but teach has not gone up that much; and second, the compensation of administrators, whose productivity should have increased many fold due to advances in IT and telecommunication, has exploded.

From Paul Campos:

One thing that rarely gets asked in the context of all this getting and spending is, What exactly is that money supposed to be for? In theory, of course, it’s for “education.” In practice, a whole lot of it goes directly into the pockets of a metastasizing cadre of university administrators, whose jobs, as nearly as I’ve been able to determine after being on a research university’s faculty for nearly a quarter-century, consist of inventing justifications for their own existence while harassing faculty to fill out evaluations of various kinds. (In a particularly Kafkaesque twist, many of these evaluations are supposed to be of the administrators’ own job performance.)

In [ex-president of Ohio State University E. Gordon] Gee’s case, the sums of money involved are disgusting. At the time he was apparently forced out after having made a few tactless jokes in a private meeting, Gee was getting paid about $2 million per year. This does not include the $7.7 million that the university paid for Gee’s travel, housing and entertainment from 2007 to 2012 — a sum that included at least $895,000 for soirees at Gee’s university-provided mansion, more than a half-million dollars for private jet travel and “$64,000 on his trademark bow ties, bow-tie cookies, O-H lapel pins and bow-tie pins for university marketing.”

Ah, yes, “marketing.”

Gee also increased the size of the university’s senior staff by 30% and raised their average salaries by 63%, to $539,390 in 2011. To get a sense of how out of control university-administrator compensation has become, consider that a year before Gee began his first tenure as Ohio State’s president, the president of Harvard was paid $138,044 ($256,000 in 2012 dollars), and only eight university presidents in the entire nation made more than $200,000. Now, thanks to Gee and his ilk, there are dozens of administrators at Ohio State University alone who would consider that sum an insult.

And for more from the OSU campus, here's Deborah J. Merritt:

We also have a problem with the salaries we pay administrators and tenured faculty. Ohio State’s Vice President for Talent, Culture, and Human Resources is earning $425,000 this year–yet she’s underpaid compared to other senior administrators. Back in 2011, they were already averaging $539,390 per year. My salary, as a senior professor in one of the best paid departments on campus, pales in comparison to these amounts–yet I know that even my salary is too high to support the goals of a public university. Ohio State’s motto is “education for citizenship.” But how much of the citizenry, and from what socioeconomic strata, are we educating at today’s prices?

(though outside of the scope of this post, Merritt also has some interesting insights into the shifting mission of schools like OSU. You should read the whole thing when you get a chance.)

And just so our readers in NYC don't feel left out, here's Pam Martens:

According to documents unearthed in a month-long search of public records, NYU Law School has created an array of nonprofits to funnel money into lavish perks for its professors. The money has been used by professors to buy multi-million dollar brownstones and condos in Manhattan and Brooklyn with portions of some loans forgiven over time. In some cases, even the interest charged on the loans has been reimbursed.

The decision to use nonprofit funds to enhance the lifestyles of a select handful of professors and administrators rather than assisting students is under investigation by Senator Chuck Grassley at the Senate’s Judiciary Committee. A referral has also been made by the NYU chapter of the American Association of University Professors to the New York State Attorney General’s Charities Bureau which oversees nonprofit organizations.

From the hundreds of records examined, NYU, under the leadership of President John Sexton, looks like a real estate developer in drag as a university. According to its federal tax returns from 2006 through 2010 – just a five year period, its five highest paid independent contractors received over $568 million for construction work and an eye-popping $173 million to clean its buildings.

Thoughts on the NYT's attempt to reboot the Airport franchise

I first heard about this through Brad DeLong who did, I will admit, have me going until I clicked through the link to this James Fallows piece. Fallows, following the lead of Gene Weingarten and Jim Romenesko*, had some questions about this dramatic account entitled "The Plane Was About to Crash. Now What?" by Noah Gallagher Shannon.

Actually, 'dramatic' is  not really an apt description. A better word would be 'melodramatic/' complete with details like

The captain came out of the cockpit and stood in the aisle. His cap dangled in one hand. “All electricity will remain off,” he said. Something about an open current and preventing a cabin fire. Confused noises spread through the cabin, but no one said a word. “I’ll yell the rest of my commands from the cockpit.” I could see sweat stains under his arms. “Not going to sugarcoat it,” he said. “We’re just going to try to land it.”

(it later came out that the plane was an Airbus 320 which is a large enough plane to raise questions about the shouting-over-your-shoulder means of communication.)

Lots of these detail raised questions, particularly with real aviation experts like Patrick Smith, who demolished the NYT piece until there was little left than a pile of phonemes, then followed up by talking to someone who actually had access to the maintenance record:

According to the information I was given, the pilots' post-flight logbook entry, which references a caution message displayed on a cockpit advisory screen of the Airbus A320, reads as follows:

"ECAM HYD Y RSVR LO LVL"

What this means, essentially, is that one of the plane's three main hydraulic systems was indicating a low level in its fluid reservoir. Airbus color-codes its hydraulic systems; this would have been the yellow (Y) system.

Per checklist instructions, the crew would turned off this system off. This is unusual, but the loss of a single hydraulic system on a modern airliner is far from a serious emergency. All critical components have at least one alternate source of hydraulic power.

Further, the corrective action note in the logbook implies the issue was merely an indication problem. Fluid quantity was found to have been at the normal level all along.

But perhaps most importantly, Shannon's essay revolves around a supposed landing gear problem. As I explained earlier (below), even the most serious landing gear malfunction sits pretty far down in the hierarchy of potential disasters. Even lower, however, is a landing gear problem that does not exist: the yellow hydraulic system does not power the landing gear.

An A320 captain I spoke to says that a shut-down of the yellow system would have meant, at worst, a slightly longer-than-normal landing roll (due to loss of the right engine thrust reverser and some of the wing spoiler panels), and, in newer A320s, loss of the nose-gear steering system, requiring a tow to the gate.

Shannon had thrown up enough red flags to begin with, but this puts it over the top, tilting his entire account from one of eye-rolling embellishment toward one of outright fabrication.

What really caught my eye, though, was the NYT editor Hugo Lindgren's response to Fallows:

Some commenters have seized on certain details of "The Plane Was About to Crash. Now What?" by Noah Gallagher Shannon in order to question whether this emergency landing happened (and perhaps even whether the author was on the flight). But there is simply no question. The author was on Frontier Airlines flight #727 on June 30, 2011, from Washington to Denver. It was an Airbus 320. The author sat in seat 12A. This flight was diverted to Philadelphia. The FAA reports that the pilot declared an emergency due to a low hydraulics indicator light and that upon landing the plane needed to be towed to the gate. Frontier airlines confirms that an Airbus A320 experienced "a maintenance issue on departure from Washington DCA. The flight diverted to Philadelphia due to easier access. The aircraft and all passengers landed safely."

Did the author's personal recollection represent an accurate picture of what he experienced on that flight? Well, only he can attest to his own experience. But the author did provide receipts and took notes after the flight to back up his account. And his recollection, when run by an aviation specialist, did seem entirely plausible to him. While some of the author's language may have been imprecise, his recollection of his experience was consistent with recollections of passengers in similar air incidents. Naturally, not every detail matches everybody else's experience. Surely even people on that plane would remember it differently. The story was about the personal experience of a fearful moment. The author did not present himself as an authority in airline technology or emergency procedures. The airline, in fact, refused his request for more information about what happened after the fact. He only reported what he heard and felt, which is consistent with the magazine's Lives page, where the account was published.

Of course, out of the tons of criticism from Fallows, Smith, Weingarten, Romenesko and company, the suggestion that Shannon had made up the flight was, at most, a tiny and generally implicit part of the case against the article. Lindgren is using some rhetorical misdirection here; he's just not doing it very well.

More troubling is the now familiar nonchalance of journalists toward accuracy. "Perceptions are subjective." "Opinions differ." "Who are we to judge whose claims are valid?"

But of they do judge. Only a select few articles make it into the New York Times. By publishing those articles, the people who run the paper are putting their reputations behind those pieces. If they want to maintain those reputations, those people need to be more careful about what they publish and less weaselly when something like this slips by.

* Make sure to check out Jeremy Repanich's comment.

Property Rights

There is an argument that has been going around that is, roughly speaking, "if Libertarianism is so great why is no country organized along these lines".  Now, it is worth noting that this comment is only true with respect to extreme or Utopian forms of Libertarianism; the movement itself has done a lot to shape a number of modern societies.  But it is true, that on first glance, this method of governance seems scarce on the ground.  Now this isn't a fatal objection, outside of Athens there were few obvious Democracies before the High Middle Ages (and the flaws of the Athenian form of Democracy are only imperfectly present in modern representative democracy). 

Mike Konzal makes the claim that Feudalism is actually a good example of Libertarianism.  Part of me wants to dismiss this claim as "too cute by half".  I also note that it immediately offends a lot of Libertarians.  And I think that is a shame as I suspect we can learn a lot from the actual analysis.

The discovery that Democracy could end up as a form of impulsive mob rule comes directly from the Athenian adventure.  Modern democrats did not reject the system because of this (potentially terrifying) flaw but rather used it to understand how to build a better system. 

So consider this argument from Konzal:

When libertarians say they are for basic rights, what they are really saying is that they are for treating what liberals consider basic rights as property rights. Basic rights receive no more, or less, protection than other property rights. You can easily give them up or bargain them away, and thus alienate yourself from them. (Meanwhile, all property rights are entirely fundamental - they can never be regulated.)

Now there may be Libertarians for which this is a feature and not a bug.  But just like modern forms of Democracy had to grapple with the issues of mob rule as seen in Athens (the death of Socrates, the insane idea of invading Sicily), understanding how this kind of system might work can actually bring out some of the key features that need to be thought through.

Two things come immediately to mind.  Property rights in the feudal era got enormously complicated, with specific places coming with extremely complex ownership rules.  People could own the right to do one thing in a certain place but not own the underlying land.  Feudal relationships were confusing, to say the least.  I think this needs to be considered.

The other issue is whether or not there should be rights in balance with property rights.  The idea that property rights are fundamental does not mean that they should be the only item in consideration.  Democracy sacrificed the purity of open voting on all matters for additional stability.  I think that these issues could be thoughtfully addressed by Libertarians looking to build a feasible model.

I do think the Randian/Objectivist ideal doesn't exist for a reason.  I can also see reasonable people not liking the core idea of property rights.  After all, unequal starting points can lead to very unequal outcomes.  But I think that this analysis should be a challenge to understanding and enhancing the operationalization of the Libertarian model.

More adventures in intellectual property --- Ghost Rider rides again

Not long ago, Marvel managed to get a couple of unlikely hit movies out of a fairly obscure motor-cycle riding character called. Ghost Rider. That unexpected success has led to some lawsuits over who owns the character.

NEW YORK (Reuters) - A comic book writer who claims he created the flaming-skulled character called Ghost Rider got another chance to press his claim on Tuesday when a U.S. appeals court revived his lawsuit against Marvel Comics.

The 2nd U.S. Circuit Court of Appeals vacated a 2011 district court ruling that found the rights to the character belonged to Marvel Comics, owned by The Walt Disney Co. It returned the case to the lower court for trial.

Former Marvel freelancer Gary Friedrich, who claims he created the character, will pursue the case "aggressively and vigorously," his lawyer Charles Kramer said.

Ghost Rider, a motorcycle-riding vigilante whose head is a flaming skull, first appeared as a vigilante superhero in 1972.

Sorta.

It turns out that the 1972 character was something of a reboot of a Western-themed Ghost Rider Marvel launched in 1967 (again with the involvement of Gary Friedrich) and that's where the intellectual property question gets amusing because that incarnation was perhaps the most blatant example of plagiarism in what has always been a plagiarism-fueled medium.

Stan Lee's claims in the Sixties that he didn't read other comic books were a lot like Milton Berle's claims that he didn't listen to other comics' material. Lee had spent two decades in a closely-knit industry jumping on trends and borrowing ideas, starting with DC and EC derived books and ending with an unacknowledged appropriation of Julius Schwartz' successful formula for rebooting Golden Age heroes with science fiction origins and streamlined looks.

The difference between Berle and Lee was that a large part of Lee's audience didn't get the joke. Many of them had gotten serious about comics in the mid-Sixties when Marvel had hit its stride and really was putting out the best and most innovative comics. For them, Lee's comically over-the-top claims seemed entirely reasonable.

It's safe to assume that in 1967, few of those teen aged fans had any idea that Marvel's new Western themed hero, Ghost Rider, had a strong resemblance to another company's hero, a resemblance that included having the same name, costume, concept, atmospheric look, and artist. The retread only ran for seven issues but that was enough for Marvel to claim ownership of the name and, a few years later, launch another Ghost Rider.

(Yes, that is a Frazetta cover)

And here's the Marvel version:

Of course, the idea to introduce a character by this name in 1949 may have had something to do with a hit song introduced in 1948 (and covered a billion times since).

When the guys who are in for murder one start arguing about Jane Austen...

This is a bit outside of our normal education beat, but this account of a man's experience teaching English and Philosophy to lifers in a British prison is definitely worth checking out.

From American Public Media's the Story.

Evolution and the breakfast landscape

Here's a question about evolutionary psychology. Not a rhetorical question or a snarky question (even though I have been a bit snarky on the subject in the past). Everyone knows why we evolved to seek out salt, sugar and fat in our diet, but why did we evolve to favor an extremely rugged yet malleable culinary landscape?

Take breakfast, for example. Sometimes (though not often) I'll hit a diner and say what-the-hell and get coffee, juice, eggs, bacon, hash browns, and biscuits. With the exception of the juice and the bacon (which are already optimized), I will add the right level of flavorings to each dish to get it to its landscape maxima (sugar for the coffee, salt and pepper for the eggs, hot sauce and ketchup for the hash browns, jam or maybe even sausage gravy for the biscuits).

The optimization is done on a dish-by-dish basis and is, for the most part, independent. If only eggs are available, I'll have a sugar-free breakfast. If all I have around the house is coffee and fruit, I'll have a low-sodium breakfast. Though there's an evolutionary imperative that makes me crave sugar and salt, it is somehow dependent on the other coordinates of the landscape. I want sugar but not salt with my coffee, salt but not sugar with my eggs and both salt and (caramelized) sugar in my hot chocolate.

Small children complain loudly and adults mumble grumpily when certain foods mix on a plate even though, as parents often remind us, it all goes to the same place.

How did our places get so context-sensitive?

Matt Yglesias may not be helping his cause

There was an odd exchange recently between Diane Ravitch and Matthew Yglesias.

Ravitch wrote a post about James Cersonsky's American Prospect article on Teach for America's political power. She introduced the post with this:

Teach for America began with a worthy goal: to supply bright, idealistic college graduates to serve in poor children in urban and rural districts.

But then it evolved into something with grand ambitions: to groom the leaders who would one day control American education.

Yglesias's response is rather strange. He doesn't mention Cersonsky or the American Prospect his post but  he only explicitly addresses points that come from Cersonsky's article; not from Ravitch. I say explicitly because the example Yglesias uses is certainly relevant to Ravitch's claim (though definitely not in the way he intended).

I thought of this over the weekend at my college reunion, where I met up with an old friend of mine who right after graduation was a science teacher in a public school in New Orleans. Later, she taught at a KIPP-affiliated school turnaround venture in New Orleans and then became founding assistant principal of a KIPP-affiliated school there. Then she moved back to the Boston area and became principal of a charter school called Excel Academy. Now she's a fellow at an nonprofit called Unlocking Potential, but soon she's going to become principal of a troubled public middle school in a a Massachusetts town whose school district has been placed in state receivership.

The part about reunion caught my eye. That's a lot of jobs for a 2003 graduate (Matt Yglesias '03 as they say at Harvard) so I did a little digging. I may have missed some important details but here's what turned up:

Barring a really astounding coincidence, Yglesias is talking about an educator named Komal Bhasin. Here's Bhasin's job history:

She taught from 2003 to 2005.

With a bachelor's degree and two years teaching experience, she was named assistant principal of a school.

With a bachelor's degree, two years teaching experience, and two years experience as an assistant principal, she was named principal of a different school halfway across the country. (You will often find sudden promotions within a school where you're dealing with known quantities. Putting a fledgling assistant principal in charge of a different school in a different region is much more unusual, particularly an administrator with almost no teaching experience.)

With these qualifications, and five years experience as a principal, she got a principal-in-residence with a high-profile education reform institute -- a relatively short tenure and thin resume for this kind of position.

Obviously, there's a limit to how much we should infer here, but Bhasin has indisputably gotten a series of promotions that were surprising given her job history, education and (as far as I can tell) publications and she has also gotten considerable exposure as a rising star in the reform movement .

Just to be clear, I am sure that Komal Bhasin is a smart and dedicated educator and may well be an excellent administrator. Nothing should take away from that, but it is also true that, given what we know, her career path would seem overwhelmingly to support the idea that she was being groomed for a Michelle Rhee type leadership role just the way Ravitch suggests.

In other words, Yglesias came up with a great example, just not for his side.

Investment advice from Alfred Hitchcock

Mail Order Prophet, a 1957 episode of Alfred Hitchcock Presents written by Robert C. Dennis (teleplay) and Antony Ferry (story) and starring E.G. Marshall and Jack Klugman

Appeared previously in You Do the Math.

"How do you make money doing this?" The answer is simple...

If they hadn't used cast members, I wonder how many viewers would have caught on?

Thoughts on Ray Fisman's much alluded-to charter school article

As mentioned before, Ray Fisman has a piece up at Slate discussing charter schools and it's quite good. Arguments for both sides are spelled out in a clear and balanced manner and, with a few exceptions (and only one of those major), data issues are addressed as thoroughly as is possible in a general interest publication.

That major issue is cheating and data manipulation and it's the one problem I would most like to get Fisman's take on, partly because I really do believe it's probably the biggest challenge facing the reform movement but mainly because of Fisman's expertise in the question of how best to set up metrics and incentives in order to keep things running smoothly.

My other issues are more the sort of technical points you probably couldn't get away with in Slate. For example, take a look at the following from Fisman:

At least part of the disagreement revolves around whether charter schools deliver on their promise to improve student outcomes. You might think this is a relatively easy proposition to evaluate—just compare whether charter school kids do better on tests than those in public schools. But any effort to compare performance is confounded by the fact that the kinds of parents who take steps to enroll their children in charter schools may be the kind of motivated and supportive parents whose children would have done just fine in any school system. (In the current study, charter school applicants do in fact have higher than average test scores even before they enroll. However, other analyses have seen charter school applicants with below-average scores, perhaps because kids struggling in the public system are more apt to look for other options.) And if the longer hours and additional school days that are a feature of many charter schools lead underperformers to drop out, the select group that remains may again be made up of those who would have tested well in any school environment.

This isn't just a question of underperformers. Based on my classroom and tutoring experience, I strongly expect we're seeing a strong interaction between treatment and selection. Different kids respond differently to different approaches. There's a very good chance that those kids (and families) who volunteer for more time and more work are also the kids who respond best to that approach. This is not an argument against charter schools; it is, however, an argument for caution in generalizing results.

A somewhat more serious issue shows up in the next paragraph

But for at least a subset of charter schools, researchers can come fairly close to running a clinical trial where some applicants are enrolled at charters and others are left in the public system purely by chance. The reason is that many charter schools are oversubscribed, and their scarce spots are allocated through a lottery. So whether a particular student gets assigned a slot at the charter school is luck of the draw. (This randomness in gaining admission to sought-after charter schools has been documented recently in the films Waiting for Superman and The Lottery.)

Numerous studies have used this lottery method to analyze the impact of charter schools on standardized test scores, and by and large they report similar findings: Charters in rural or suburban areas don’t do any better than public schools, while in urban areas they are associated with greater test score improvements in math and language. But another important point from past studies is that there is enormous variation in the effectiveness of charter schools. There are some great ones but also some real duds.

Normally, in clinical trials, we can treat our subjects as independent. After all, my odds of developing a side effect should have nothing to do with whether or not you developed that side effect. There is simply no way you can assume that the behavior of one student is independent of the rest of the class and that lack of independence can be extraordinarily difficult to account for. While useful, the lottery method is a long way from giving us the results we would expect from a good clinical trial.

But, that aside, Fisman's point about urban schools is tremendously important and is explored in more detail later:

Some charter school advocates will surely point to the new study as yet more evidence that public school districts should be replaced by a more decentralized approach to education, with a greater emphasis on charter schools. But that’s a very narrow interpretation of the mounting evidence of charter schools’ successes. Focusing on these successes glosses over the many cases where charter schools fail to outperform their public peers. For suburban districts in Massachusetts, for example, the numbers don’t favor the charter school advocates, in large part because suburban public schools in the state are pretty effective already.

This is a huge point and it addresses one of the biggest concerns of reform skeptics, namely that real problems might be used as cover to change the parts of the system that are working but which are ideologically disagreeable to some of the reformers and, as a result, leave a system more broken than it is now.

From the beginning of the debate, the question of how to close the achievement gap has gotten conflated with the question of general education reform. The wretched state of education in high poverty areas was often used to argue that we needed to make immediate sweeping changes to all areas of education and some of those changes were truly radical.

Until recently, many of the reform proposals (firing 80% of new teachers, extensive privatization, effectively removing teachers' right to unionize, and quite a few more) allowed for little if any middle ground, particularly given the amount of data that was not breaking in the reformers' favor. I strongly suspect that Diane Ravitch's Road-to-Damascus conversion came from her inability to move gradually away from her previous positions.

I realize I'm reading a lot into one recent article, but this seems to be where serious reformers like Fisman and Jonah Rockoff are now:

1. Charter schools have a great deal of promise;

but

2. They aren't right for all kids or all situations;

3. and there may be scaling limitations both in terms of teachers and parents;

4. so we should focus on areas where the need is greatest and the evidence for effectiveness is strongest.

This is not the exact list I would have made (OK, technically I did make this list but you know what I mean), still there's nothing here that I would disagree with. What we have now are questions of degree, which tend to be more productive and are much more likely to be convergent.

Which would be nice for a change.

Give my regards to bean town, but...

I found quite a bit to agree with in Ray Fisman's recent piece on charter schools (and even more to like about it), but I do have a couple of issues I wanted to get out of the way. The first involved the threat to reform presented by people who cheat and/or game the system. The second revolved around some of the issues with the heavy reliance on a small set of Boston schools as examples.

I haven't had time to go through the Boston report in detail, but assuming that it's solid (and I have no reason to think otherwise), there are still a couple of reasons to be careful about drawing overly broad inferences from this particular city at this particular point in history.

The first (which is pretty much self-evident) has to do with Boston's unique relationship with higher education. The second, and more important, goes to the heart of the problems with modern urban schools.

Here's Fisman:

Why wouldn’t parents try to get their children into charters that would get the kids on track to go to college? One explanation is that the debate about whether charter schools “work,” with its focus on testing and college placement, loses sight of the many reasons why people choose a school and what they value in an education. Charter schools can be set up for any purpose. Some focus on the arts, others emphasize cultural heritage (there are multiple Hmong charter schools in the Twin Cities alone); some are vocational, others rigorously academic. As Rockoff puts it, asking whether a particular school is good based on test scores or college placement is in many cases the wrong question: “To extend the restaurant metaphor, some people like Italian, others like Thai food. Similarly, many [charter] schools focus on tested subjects, while others might emphasize creative writing or the arts.” Not every charter school is right for every kid. Nor would it be an unmitigated good if all parents could segregate their children by their interests and beliefs.

But a big reason, maybe the big reason, that urban parents send their children to charter/magnet and parochial schools is left out of that paragraph. In general, parents want their children to be successful happy and safe, but of those, safety is the most immediate. These non-standard schools usually have a low or no-tolerance approach to gangs. More significantly, there's a huge anti-gang selection effect.

If you want to compare different urban education initiatives, you have to take into account gang activity and neighborhood violence, not just in terms of absolute levels but in terms of attitudes and expectations, and, while I have no expertise in the area, I do know enough to know that understanding urban crime in Boston over the past twenty years, a period that saw both the advent and apparent collapse of the "Boston Miracle," is something that does require considerable expertise.

None of this undercuts the findings of the study, but it does greatly complicate the process of drawing inferences from it. It's possible that what worked in Boston won't work as well in most urban areas. It's also possible that what worked there will work even better elsewhere. I'm not being cute here -- I could honestly see this going either way -- I'm just trying to emphasis just how messy this data is.

We have here everything your stat professor warned you about. Interactivity. Non-linearity. Heteroskedasticity. Nesting. Selection effects. Missing not at random. Falsification. Hawthorne effects. Volunteer effects. Peer effects. Researcher bias. Sample size issues.

Though I'm not a Bayesian by training, I think the appropriate response is at least in spirit, Bayesian. We will have to factor our doubts about the research in when we weigh our conclusions but that's not the same as ignoring those conclusions entirely.

This brings me back to what I liked about Fisman's article. It's balanced, it acknowledges the limitations of the data, and it suggests focus on areas where reforms will do the most good if their assumptions are sound and will do the least damage if they aren't.

But more on that later.

Two things you should read in their entirety

The first is Mike Konczal's analysis of the drivers of mass incarceration, "Against Law, For Order"

As Bernard Harcourt examines in The Illusion of Order, broken windows policing is predicated on separating neighborhoods into regular, ordered insiders and disordered strangers. [James Q.] Wilson’s view is that regular insiders are the “decent folks” who need to be protected from the disorder generated by strangers. The police, rather than upholding laws and the rights of citizens, uphold order by regulating the behaviors of disorderly insiders and excluding the disorderly outsiders. Criminals lose their insider status in this telling, and excluding them from the community becomes a goal of law. The approach is based on a privileging of order over law, for a lack of order is what attracts criminal behavior, always waiting in the wings to descend.

The second is this short piece from TPM.

A Texas man, Ezekiel Gilbert, has been acquitted of murder after shooting and killing a 23 year old escort he found on craigslist because she would not have sex with him even after paying her $150 fee.

Gilbert was acquitted under a Texas law that allows you to use deadly force to protect your property during a nighttime theft. In this case, the property was either the $150 or the use-rights to the body of 23 year old Lenora Ivie Frago. Frago said she couldn’t give back the money because she had to give it to the driver who was waiting for her.

At the risk of putting too fine a point on this, these deadly force laws have always been about keeping people in their place, socially, economically, racially. Crimes against people who rank low on the hierarchy are seen as less heinous in this Wilson-like view and Ms. Frago ranked low in the eyes of the jury. I think it's safe to say that if a prostitute had killed a man in Texas because he had refused to pay after sex, she would certainly be in prison and probably be on death row.

Contests big and small

This is a follow up of sorts to the recent Project Gutenberg Project post, not in that it involves similar data or statistical techniques but in that both are part of a loose thread on the possible (and in some cases, inevitable) tranformative movements in statistics and research that are generating a lot of talk but perhaps not enough discussion.

Last time I had a post on how we might want to open big data. This time the focus is big contests. Things like the X-Prizes or proposals to offer awards for the development of new drugs. The qualifier 'big' is important; scale plays a significant role in the second half of the post and I'll get to some smaller contests then. For now though, let's consider big contests for huge organizations.

If you put aside reputational capital (which might be a big deal to a small start-up but which wouldn't have much impact on a big multinational), in order for contests to make sense, you're left with the following two inequalities:

Existing incentives < Cost of development

but

Existing incentives + (Prize * Probability of winning) > Cost of development + Uncertainty penalty

That's the picture for the competitors and it's hard to imagine them being satisfied that often.

What about the party offering the prize? For them, much of the benefit comes from the number of groups competing for the prize; presumably the quality of the winning entry should increase monotonically as you add competitors. Number of entries is, of course, inversely related to cost of development.

These contests have long had a special appeal for freshwater economists, but when you get past the open enrollment aspect, the process looks awfully bureaucratic. The competitions are driven by artificial criteria determined by a committee. What's more the criteria generally only have to be met once during a relatively short time frame rather than over the lifetime of the product. This produces a great incentive to cheat.

If you were to sit down and, given these factors, try to come up with the worst possible product to develop with a contest it would probably be in pharmaceuticals. Drug development has huge costs, is largely limited to a small pool of players and is highly vulnerable to fraud (fraud which comes with substantial social costs). Despite this drugs are one of the most frequently proposed applications of the contest model.

The various X-prizes are fortunately not vulnerable to fraud, but they are also poor choices for the contest model: massive projects (though, with the exception of the tricorder, not necessarily that ambitious from a technological standpoint) requiring huge start-up costs that can be undertaken by only a handful of competitors (think dozens instead of hundreds).

The contest model can, however, make a great deal of sense assuming you can meet the following conditions:

A relatively large pool of potential competitors;

Relatively low cost of development;

A relatively high secondary reward/cost of development ratio (secondary rewards include reputation, access to future contracts, publications and for students, grades and thesis material);

Largely ungameable criteria;

Significant positive externalities (for example, a text mining competition might increase the competitors' programming skills and thus increase the skilled labor pool).

For example, DARPA's Grand Challenge contests met all of these conditions. Hundreds of teams competed in the various events and almost certainly produced more quality research and innovative thinking than we would have gotten spending the same money in a straight grant arrangement.

Here, however we run into a different question of scale, not in participants but in number of projects. The Grand Challenge worked not only because it met the conditions listed above but because it took advantage of a certain slack in the system. There were enough people in engineering and computer science departments who were looking for an interesting, high-profile project and who didn't care that the expected value of the compensation didn't come close to paying for the time, facilities and equipment required to compete. How many of these competitions can we introduce before we start cannibalizing the talent pool?

When done badly, contest-based research makes little sense, but even when done well, it can't really be more than a small part of our overall research strategy.