Comments, observations and thoughts from two bloggers on applied statistics, higher education and epidemiology. Joseph is an associate professor. Mark is a professional statistician and former math teacher.
One of the nice things about the internet is that, if you look hard enough, you can find people addressing important questions about where statistics (in this case, the plural of statistic) come from.
Over at MacroMania (via Thoma), David Andolfatto talks about how we measure unemployment:
So this is how it works. You are asked in a survey whether you have done any paid work in the previous four weeks. If you answer yes, then you are counted as employed. If you answer no, then you are nonemployed.
If you are nonemployed, you are asked another question: Have you been doing anything in the past four weeks to find work (and if so, what)? If your job search activity is judged to be sufficiently "active," then you are counted as unemployed; otherwise, you are counted as a nonparticipant.
This suggests an interesting point. It would appear that when we talk about job seekers (as in the ratio of job seekers to job openings), we may be excluding the severely underemployed which means things are even worse than they look.
Is this something we should worry about? Damned if I know. That's a question for an econometrician. But it is a good question to ask.
Another good question is who decides on categories and definitions get used when putting together statistics. There have been a lot of pixels spilled about this hypothetical taxpayer receipt put out by the Third Way think tank, but so far as I know, Andrew Gelman was the first to notice this:
I'm more concerned about the list itself, though. I think a lot of cognitive-perceptual effects are involved in what gets a separate line item, and what doesn't. For example, I see the FBI but not the CIA, the NSA, or weapons procurement. There's a line for "salary and benefits for members of Congress" but nothing for the courts system or the White House. And so on. So, while I agree with [Megan] McArdle that "more information is generally better," I'm not quite sure how to get there. I'd be very very suspicious of the choice of items that happens to end up included on the hypothetical itemized tax bill. Especially If it's really true that people don't notice those boxes on their W-2 form with FICA and Medicare payments, I also seem to recall seeing some glossy government documents with charts showing where the money is coming from and where it goes. Maybe there's some place other than a W-2 form to put this information where people will notice it.
Tell everybody we have a winner (from the WSJ.com via Thoma):
HOUSTON—At midnight on the first of the month, a scene unfolds at many Wal-Mart Stores Inc. sites that underscores the deep financial strains that many low-income American consumers still face.
Parking lots come to life after 11 p.m. as customers start to stream into the stores, cramming their shopping carts full of milk, infant formula and other necessities.
Then at midnight, when the government replenishes their electronic-benefit accounts with their monthly allotments of food stamps, nutritional grants for mothers with babies or other aid for needy families, they head for the registers.
Felix Salmon follows up on his airport post and actually makes matters worse. He asks an expert (Greg Lindsay) what it is that makes the Hong Kong airport good and the response is efficiency. These airports are able to move people quickly and effectively (which, in the end, makes the travel experience more pleasant). But he concludes:
So I’m still not convinced that a major investment in airports is the best — or even a modestly good — use of federal infrastructure-investment funds. Yes, America’s airports are miserable places to travel through. But if what we want to do is boost long-term GDP, then there are better places for the government to spend its money. As and when airports get replaced and upgraded, they will naturally become more modern and efficient. Sadly, however, that’ll take time — and it might not make the passenger experience much better.
I wonder, in a time of low aggregate demand, why investing in more efficient airports is a less favorable form of economic stimulus than building roads or bridges? After all, construction labor is currently cheap and underutilized. Refitting airports so that TSA screening creates fewer queuing problems (for example) or to make them less vulnerable to delays due to poorly designed runways hardly seems like a bad use of infrastructure funds. And it will improve the passenger experience, if only indirectly by reducing the negative externality of lost time due to queuing issues (both of planes and people) that is currently imposed on passengers.
What is the comparison infrastructure investment that is clearly superior? I can think of a few transportation based ones (like improving public transport) but these often lead to new operating costs (running a new bus service, for example). Making airports more efficient might result in lower future costs due to more modern facilities and a better functional design.
In response to a widely circulated itemized breakdown of where your tax dollars go, McArdle writes:
There seems to be an unspoken assumption that opposition to spending rests on misperception of what the money is spent on; Americans tell pollsters they want to cut spending, but it turns out that what they really want cut is the imaginary fortune they think we spend on foreign aid.
But of course, it seems to me that this could just as easily go the other way: isn't it possible that the widespread support for programs like Social Security and Medicare rests on the fact that most people don't realize just how big a portion of your paycheck those programs consume? I don't know the answer to that, but I will point out that most economists believe that paycheck withholdings enable (among other things) higher taxes; if people had to write out a check for their tax bill every year, resistance to income tax increases would be much fiercer.
This suggests that handing people an itemized invoice for their government programs which shows them the total yearly take might increase support for the Smithsonian, and decrease it for the stuff that appears higher up on the bill.
Whichever way it cuts, I think this is a good idea; more information is generally better. I just think that the emotions this sort of receipt provokes in liberal bloggers may not be the same ones it provokes in the average voter.
I think this is a great idea and I'd like to see it implemented immediately. Is it too late to add boxes to our 2010 W2s for Social Security Tax Withheld and Medicare Tax Withheld?
This series of post (go here, here, here, here, and while you're at it, here) got me thinking about a rhetorical trick that has gained great popularity in the op-ed pages and other places you find pundits at work. It's called the RMP/STF argument and it's something you can try at home.
Start with a Relatively Minor Problem (RMP) usually expressed through a vaguely troubling statistic. Here are some RMPs:
We might eventually be forced to cut Social Security benefits though not by that much and not for a long time;
A test of international academic performance shows America near the bottom though the test is fairly new and other better established tests and studies contradict this finding;
Airports lack amenities (like plug-ins for laptops) and it can be difficult getting from one terminal to another.
Now pair your RMP with a Serious Tangentially-related Fact (STF). Here are some examples of STFs:
If current trends continue, Medicare will be unsustainable in the near future;
TSA procedures of questionable value have a large cost to the economy in terms of lost productivity and tourism dollars.
Finally, you propose a solution to your original RMP:
Privatize Social Security;
Fire teachers who don't produce higher test scores;
Reduce zoning restrictions to create more meaningful competition for airports.
Of course, privatizing SS does nothing to help Medicare, reducing zoning restrictions will have no effect on TSA rules and focusing on test scores actually gives schools an incentive to encourage students to drop out, but that doesn't matter. The argument works on an emotional level. It uses the negative associations of the STF to make people more receptive to the solution.
This is closely related to Paul Krugman's "serious person" phenomena where economists, politicians, bankers, etc. are considered responsible and realistic for taking positions directly contradicted by the data. Once again it is the emotional association that matters.
When you criticize an RMP/STF argument, someone will invariably say "At least, _____ is willing to talk about [serious issue]." That's not really true. _____ is willing to invoke the issue and we need a higher standard than that.
Alex has a great link to a really interesting series of posts on United States health care costs. Just reading over the slides that they so nicely provide gave a very nice introduction to some of the most important policy questions of the day.
Q. What precautions will be taken to insure that there is no terrorist bomb aboard my aircraft? A. The airline agent will ask you a series of security questions shrewdly designed to outwit terrorism, such as: "Did any terrorist unknown to you give you a bomb to carry on board this plane?" Also, if you have a laptop computer, they may ask you to turn it on, thus proving that it is not a terrorist bomb.
Q. But couldn't a terrorist easily put a bomb in a computer in such a way that the computer could still be turned on? A. Shut up.
Prepare for a rare intra-blog dispute. Joseph found something to like in Megan McArdle's reply to Felix Salmon's post on airports. Here's the relevant section from the Atlantic:
Still, I think there's quite a lot about American airports that is important, and inadequate. Given the ubiquity of electronic devices, and the importance of airports to business travelers, we could probably enhance national productivity quite a bit if so many airports didn't force travelers to spend their wait times fighting each other for the one electrical socket located behind an out-of-order ATM machine. The ridiculous security theater procedures which have queues stretching out towards the long-term parking lot could be streamlined. And whatever engineer designs monstrosities like Heathrow's 40-minute walk-time from security line to gate should be tracked down and . . . um . . . reeducated, or something.
We might also give serious thought to whether something can be done about the incentives system--and local authorities--who fix things so that the only important customers of airports are the airlines. In many places, a combination of zoning, and the local authorities who often run the airports, means that there's no meaningful competition. The result is that they don't have to do anything to please passengers, and boy, they sure don't. If Felix's point is that improving the airports is probably not going to be a matter of huge government expenditures--or that this is not the best use of said expenditures--I'm pretty sure I agree. But we might think about regulatory changes that would give them reasons to improve.
For starters, I can't award any points here for criticizing the TSA. Everybody hates the TSA. Joseph does. I do. Felix Salmon does. Pretty much anybody who's been in an airplane in the past nine years does. You can't give a writer credit for voicing a universal opinion.
What's left?
A couple of complaints about airport amenities and some wonderfully McArdlesque notions about how market forces work.
It's certainly true that airports could provide a more pleasant experience for passengers but, as Felix Salmon points out, "[T]he airlines are the customers, and the passengers are the goods being transported." This isn't a case of misalligned incentives; this is the business model.
As far as I can tell, the Atlantic post doesn't propose an alternate business plan or a slate of restrictive airport regulations (which are far more difficult to justify than additional regulations for flying). Instead you get a classic McArdle action plan:
Step One -- Relax zoning and other restrictions.
Step Two --
Step Three -- Traveler's paradise.
Just how far apart are steps one and three?
Even with the laxest possible restrictions, there are practical limits on where you can put airports. For one thing it's kind of important to space them out. Would a change in zoning rules add enough airports to create meaningful competition? Almost certainly not.
And even if it did, that would do nothing, absolutely nothing, to reform the more absurd TSA policies. An improvement in airports that doesn't address the security procedures is a damned small improvement.
What's worse, there's reason to believe that even those small improvements wouldn't come to pass. Under the current business model, the only leverage passengers have is the option of voting with their feet, avoiding an airport in such large numbers that the airlines pressure that airport to improve.
Unfortunately, this runs into the same ugly business reality that airlines have encountered when trying to attract customers with all-class amenities. When you go online to book a flight you are normally presented with a handful of choices that vary widely in price* and convenience. The variation in these factors tends to swamp everything else. Factors like a more comfortable plane or better customer service only come into play in the case of a very close tie. This is one reason why most airlines moved away from pushing amenities and focused on loyalty rewards programs.**
Airlines are far more competitive than airports will ever be and most people value a comfortable flight much more than a comfortable airport. If market forces haven't made flights pleasant, they aren't going to do any better with the places we wait for them.
Q. Airline fares are very confusing. How, exactly, does the airline determine the price of my ticket? A. Many cost factors are involved in flying an airplane from Point A to Point B, including distance, passenger load, whether each pilot will get his own pilot hat or they're going to share, and whether Point B has a runway.
Q. So the airlines use these cost factors to calculate a rational price for my ticket? A. No. That is determined by Rudy the Fare Chicken, who decides the price of each ticket individually by pecking on a computer keyboard sprinkled with corn. If an airline agent tells you that they're having "computer problems, " this means that Rudy is sick, and technicians are trying to activate the backup system, Conrad the Fare Hamster.
** Because most people tend to go to the closest airport, an airport loyalty program would spend most of its budget on customers it would have gotten anyway. Bad idea.
It is not everyday that I agree with Megan McArdle over Felix Salmon. But this is an extremely good point:
Still, I think there's quite a lot about American airports that is important, and inadequate. Given the ubiquity of electronic devices, and the importance of airports to business travelers, we could probably enhance national productivity quite a bit if so many airports didn't force travelers to spend their wait times fighting each other for the one electrical socket located behind an out-of-order ATM machine. The ridiculous security theater procedures which have queues stretching out towards the long-term parking lot could be streamlined.
To be blunt, the modern American airport seems designed to make a basically unpleasant activity (flying around in a packed airplane) as unpleasant as possible. Your humble narrator has been doing a lot of travel lately and I remember airports as being more pleasant once. For example, when you did not have to go through a lengthy screening process then you did not have to arrive as early at the airport. As a result, the airport had fewer bored people sitting around competing for limited seating and eating facilities.
So I would also be in favor of finding ways to make it easier for airports to make flying a pleasant experience.
From the Boston Globe by way of Mippyville, here's some mildly warped satire to end your day. Not as funny as it might have been but lovingly executed.
(depending on your browser, you might need to click on the image to see the fourth panel.)
Richard D. Kahlenberg, editor of the forthcoming book "Affirmative Action for the Rich: Legacy Preferences in College Admissions," points out that universities in other countries don't give so-called legacy preferences to sons and daughters of their alumni. (Even Oxbridge colleges don't, despite the class-bound history of British education.) So, he asks, why on earth do we do it in America?
Broadly speaking, students go to college in search of four things: certification; instruction; reputation; and connections.
In terms of certification, any well-accredited school would do. In terms of undergraduate instruction, the best deal for the money (and perhaps the best deal period) is the small four-year school. (I'm leaving this as an assertion but I'm fairly confident I can argue the point if anyone wants to debate.)
In the next two categories, however, the Ivy League cannot be surpassed, in part because of the legacy system.
Without loss of generality, look at Harvard. The student population of the school consists entirely of two overlapping groups: people who can get into Harvard; people whose parents can get them into Harvard.
The first group is hard-working, ambitious and academically gifted. Assuming the number of need-based legacies is trivial, the second group comes from families that are wealthy (they're paying for a Harvard education) and well-connected (at least one parent went to Harvard).
Putting aside luck, you can put the drivers of success into three general categories: attitude, drive and work habits; talent, intelligence and creativity; reputation and connections. It is possible to succeed with just one of these (hell, I can think of people who made it with none), but there is a strong synergistic effect. A moderate talent who works hard and has connections will generally go farther than a spectacular talent who's lazy and isolated.
Connections are governed by the laws of graph theory. I'm not going to delve too deeply into the subject (since that would require research and possibly actual work on my part), but as anyone who has read even the cover blurbs on Linked or Small Worlds can tell you, adding a few highly connected nodes (let's call them senator's sons) can greatly increase the connectivity of a system.
It would be interesting to model the trade off between picking a well connected legacy over a smarter, harder-working applicant given the objective of producing the greatest aggregate success. Because of the network properties mentioned above, it wouldn't be surprising if the optimal number of legacies turned out to be the 10% to 15% we generally see.
Optimized or not, this mixture is almost guaranteed to churn out fantastically successful graduates regardless of what the schools do after the students are admitted. I'm certain the quality of instruction on the Ivy League schools is very good, but, like most education success stories, the secret here is mostly selection and peer effects.
Update: For a different interpretation (this time with actual data), check out this post at Gene Expression.
Updated update: Why doesn't spell check work in the title field?
The recent debate on education has been pretty broad and I wonder if a key point has been overlooked. In the cross country comparisons, Mark has been arguing that "even if we grant that cross country comparisons are useful, they don't necessarily say what you think they say".
This has led to overlooking the key problem in these comparisons: it is nearly impossible to relate the performance of a specific country to that of individual students because the make-up of the students differ. The real question we want to ask is: if we adopted the educational system of country X then would our students do better? But that the student in country X do better than Americans is no proof -- maybe they are handicapped by an inferior educational system and would do even better in an American style educational environment.
This problem is a key limitation in epidemiological studies of all kinds. If we observe that the Japanese have fewer myocardial infarcts then we still don't know why. There are many exposures that could explain this finding. It is easy to go wrong. For example, in 2002-2003 47% of Japanese men smoking and 20% of American men smoked.
Yet in 2002, the age standardized death rate for American men from CHD was 216 per 100,000 whereas it was 54 per 100,000 in Japan. (now this association is cross sectional but most cross country educational debates are as well). Does this mean we should promote smoking to reduce heart disease?
If the answer is "no" then we should apply similar caution to other international comparisons.
You gotta love the headline writer for the New York Times:
"Republicans’ Deficit-Cut Pledge Lacks Specifics"
Over on the op-ed page* Paul Krugman is a little more blunt:
"‘Pledge to America’ is at war with arithmetic"
Here are some of Krugman's specifics:
True, the document talks about the need to cut spending. But as far as I can see, there’s only one specific cut proposed — canceling the rest of the Troubled Asset Relief Program, which Republicans claim (implausibly) would save $16 billion. That’s less than half of 1 percent of the budget cost of those tax cuts. As for the rest, everything must be cut, in ways not specified — “except for common-sense exceptions for seniors, veterans, and our troops.” In other words, Social Security, Medicare and the defense budget are off-limits.
So what’s left? Howard Gleckman of the nonpartisan Tax Policy Center has done the math. As he points out, the only way to balance the budget by 2020, while simultaneously (a) making the Bush tax cuts permanent and (b) protecting all the programs Republicans say they won’t cut, is to completely abolish the rest of the federal government: “No more national parks, no more Small Business Administration loans, no more export subsidies, no more National Institutes of Health. No more Medicaid (one-third of its budget pays for long-term care for our parents and others with disabilities). No more child health or child nutrition programs. No more highway construction. No more homeland security. Oh, and no more Congress.”
* I could have linked this directly to the New York Times, but frankly they annoy me.
p.s. Keep an eye on TNR and Economist's View for reactions to the rest of David Leonhardt's article. When Chait, Krugman, DeLong and company get to Leonhardt's defense of Paul Ryan and Rand Paul, things are going to get bloody.
We've recently had a huge uptick in viewers. This is great (and I'm tremendously appreciative), but I do have to remind myself most of the people in the room walked in on the middle of the conversation.
If you do a keyword search, you would see we've been hammering away at Ms. Darby for a long time and you might get the impression that we are trying to depict her as the mad relative chained up in the New Republic's attic. We're not and she isn't. (TNR's attic position was filled long before she got there.)
Ms. Darby, a talented and well-qualified journalist, takes the brunt of the criticism because she writes the bulk of TNR's articles on education, not because those articles deviate from the magazine's position on, well, anything.
The indispensable Jonathan Cohn has handed his blog over to Darby's education posts while Jonathan Chait, arguably the best political blogger out there, has taken an almost identical position on the subject.
I addressed Chait's uncharacteristic education writing a couple of weeks ago (seems like longer) in "Strange Bedfellows":
There's too much here to cover in one post (I could do a page just on Chait's weird reaction to Samuelson's looks, a topic that I had never given any thought to up until now). I may take another pass at another section later but for now I'm going to limit myself to this particularly egregious bit:
How does Samuelson explain the existence of new charter schools that produce dramatically higher results among these lazy, no-good teenagers? He insists, "no one has yet discovered transformative changes in curriculum or pedagogy, especially for inner-city schools, that are (in business lingo) 'scalable.'" This is utterly false. The most prominent example is the Kipp schools, which have shown revolutionary improvements among poor, inner-city students and have rapidly expanded.
It is strange to see Chait take the pro-privatization side of the debate, stranger still to see him accuse critics of charter schools of having an anti-government bias*, but what pushes this into Rod Serling territory is the spectacle of having Chait, one of the most gifted bullshit detectors of the Twenty-first Century, rolling out the same sort of flawed argument that he has made a career out of dismantling.
In order to be viable, a reform has to improve on the existing system by a large enough margin to justify its implementation costs, but if you accept the metrics used by the reform movement, then you will have to conclude that charter schools do worse than public schools more often than they do better.**
So we have a major push to privatize government services which, after about two decades of testing have been shown to under-perform their traditional government-run alternatives. Rather than show why this statistic is misleading, Chait pulls out vague, anecdotal evidence of a single out-lier. Now, given the variability of the data, we would expect the top schools (or even chains) to do pretty well. That alone rebuts Chait's point, but it gets worse. Self-selection, peer effects and selective attrition*** all artificially inflate KIPP's results. When you take these factors into account, it's hard to make a compelling statistical case that even the best charter schools are outperforming public schools (though the second footnote still applies).
At the risk of over-emphasizing, this is Jonathan -- freaking -- Chait we're talking about, a writer known for his truly exceptional gift for constructing logical arguments and, more importantly, spotting the fallacies in the arguments of others. Under normal conditions, Chait would never fall for a badly presented argument-by-anomally, let alone make one, just as, under normal circumstances, a confrontation between Samuelson and Chait would result in little pieces of the former being scraped off of the walls of the Washington Post.
But Chait loses this confrontation decisively. From his ad hominem opening to his factually challenged close he fails to score a single point. And this is far from the only example of this odd reform-specific impairment affecting otherwise accomplished writers. OE has spilled endless pixels on the reform-related lapses, both statistical and rhetorical, of smart, serious, dedicated people like Chait, Seyward Darby and, of course, Ray Fisman (just do a keyword search). None of these people would normally produce the kind of work we've cataloged here. None of them would normally ignore the defection of one of the founding members of the reform movement. None of these people would normally feel comfortable dismissing without comment contradictory findings from EPI, Donald Rubin and the Rand Institute.
David Warsh has aptly made the following comparison:
Remember the recipe for a policy disaster? Start with a handful of policy intellectuals confronting a stubborn problem, in love with a Big Idea. Fold in a bunch of ambitious Ivy League kids who don’t speak the local language. Churn up enthusiasm for the program in the gullible national press – and get ready for a decade of really bad news. Take a look at David Halberstam’s Vietnam classic The Best and the Brightest, if you need to refresh your memory. Or just think back on the run-up to the war in Iraq.
but along with Halberstam, it might be time to brush off our copies of Cialdini's Influence.
From a data standpoint, the past few years have been rough on the reform movement. Charter schools have been shown to be more likely to under-perform than to outperform. Joel Klein's spectacular record turned out to be the product of creative accounting (New York City schools have actually done much worse than the rest of the state). Findings contradicting the fundamental tenets of the movement accumulated. Major figures in research (Rubin) and education (Ravitch) have publicly questioned the viability of proposed reforms.
As Cialdini lays out in great detail, when you challenge people's deeply held beliefs with convincing evidence, you usually get one of two responses. Sometimes you will actually manage to win them over. More often, though, they will dig in, embrace their beliefs more firmly and find new ways to justify them.
I think it's safe to say we don't have response number one.
* Almost all of the major tenets of the modern reform can be traced back to the Reagan era and were closely associated with the initiatives described in Franks' The Wrecking Crew.
** Ironically, if you consider the intellectual framework of the reform movement to be flawed and overly simplistic, you can actually make a much better case for charter schools.
*** From Wikipedia: "In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."
