Causal Inference

Andrew Gelman comments on an article linking genetic diversity (both high and low) with less economic performance than countries with middling levels of diversity.  His take-away is quite good:

High-profile social science research aims for proof, not for understanding—and that’s a problem. The incentives favor bold thinking and innovative analysis, and that part is great. But the incentives also favor silly causal claims. In many social sciences, it’s not enough to notice an interesting pattern and explore it (as we did in our Red State Blue State book). Instead, you’re supposed to make a strong causal claim even in a context where it makes little sense.
 

But I also think it omits one piece that is crucial for causal claims: what does a counterfactual look like?  This happens a lot with complex phenomenon in both medicine and social science.  Just look at the question of whether or not to adjust for variables like blood pressure and cholesterol when estimating the effect of obesity on mortality:

 It's possible that most of the thin people who die are meth addicts or have cancer, but even a study which threw out the folks who died within three years of entry into the study found that once you accounted for physical activity*, "underweight" BMIs were correlated with excess mortality risk, while "overweight" BMIs were not. And arguing that the study fails to control for things like blood pressure, blood sugar, and cholesterol seems like fairly weak sauce; those are the very mechanisms by which obesity is supposed to kill us. 
 

So what would it mean to make a person thinner and not influence the mediating factors through which the disease operates?  It would be a thin person with a lot higher risk of mortality, I suspect.  It's the same example as imagining an antihypertensive medication conditioned on blood pressure -- one would suspect that the causal effect of the drug on the participant would be different if it failed in its primary function. 

In the same sense, the question of how to change genetic diversity without influencing a lot of other variables is a tricky one.  What would it mean for a country whose genetic composition was unrelated to migration to change their level of diversity without changing other factors?  What is the mechanism by which we think this operates?  Mechanisms are not very important for randomized trials because the design eliminates confounding.  But for a non-randomized study, this is a very important piece.

And if we argue that this is just a proxy variables (which seems to be the route that Andrew is taking in his discussion)  then the hard causal claims are unecessary.  Even worse, they may well obscure factors on which we could imagine basing a strong counter-factual.  Exploring data like this is an extremely interesting exercise but I agree that I wish we could admit when we see an interesting pattern that we may not know why this pattern exists. 

Professors and stress

This reply to the classic Forbes article by Susan Adams is worth reading. My favorite piece:

Write a grant application, get three anonymous reviewer critiques. Submit research results for publication in a peer-reviewed journal, get anonymous reviewer critiques. Submit your tenure portfolio or post-tenure portfolio to a college promotion and tenure committee, get anonymous reviews. While one may know the general composition of grant review and promotion and tenure committees, you don’t know precisely who is gunning for you. Anonymity is sometimes useful but more often allows petty vendettas to occur that are independent of the work at hand.
 

It is amazing how true this can be and how hard it is to try and modify your approach based on feedback when the next set of anoymous reviewers could be completely different. 

Metrics on education

Matt Yglesias has a post up about the StudentsFirst report claiming that outcomes are complicated to measure.  While this point is, in the abstract, true, it is informative to see what the highest ranked state (Louisiana) does in the grade 8 reading tests results that he shows.  He gives a number of groups: Everyone, Africian American, Latino, Low income, and White.  Louisiana is in the below average group for all groups except Latinos (where they are average).  It is also worth noting that Latinos make up about 3% of Louisiana's population, whereas African Americans make us around 34% of the population.  So they get average results only in a very tiny minority population.

Now there are lot's of reasons why a school system might be doing the best that they can and student results are divorced from a lot of complex social phenomenon.  However, when the top ranked states is below average on most student outcomes and above average for no populations that should be concerning. 

Now maybe the reforms have been too recent to have results.  But if reforms have very long lag times then we have another problem -- how do we properly evaluate the quality of reforms if it takes a decade to be seen in the test results? 

UPDATE: See here for the actual correlation coefficients.  Consider:

Looking more rigorously at the results, the correlation coefficient on the rankings in the StudentsFirst report card with state rankings on reading scores is -0.20. (The correlation coefficient is a measure of the similarity of two sets of numbers, ranging from -1.0, completely dissimilar, to +1.0, perfect similarity.) That’s not a large number, but the negative sign means that the correlation is in the wrong direction: the higher the StudentsFirst score, the lower the NAEP reading score. The correlation on math is even worse, -0.25.

It's not a good sign when the outcomes data is in the wrong direction. 

EDIT 2: Missing link inserted

Charter school tricks: an ongoing saga

This is remarkable:

Boston’s Commonwealth charter schools have significantly weak “promoting power,” that is, the number of seniors is routinely below 60 percent of the freshmen enrolled four years earlier. looking at it another way, for every five freshmen enrolled in Boston’s charter high schools in the fall of 2008 there were only two seniors: Senior enrollment was 42 percent of freshmen enrollment. in contrast, for every five freshmen enrolled in the Boston Public Schools that fall there were four seniors: Senior enrollment was 81 percent of freshmen enrollment.

High graduation rates seem to be misleading if the weaker students are simply being pushed out and back into the public system (or even worse not in the system at all).  An honest conversation about choice requires that we be aware of the ways that private institutions are different than public ones.  I know people who have had their kids kicked out of a daycare because it wasn't working out and because a private institution can do what it wants with customers.  The ability to remove disruptive students is certainly a nice benefit, but does the likely arms race really work out for the children involved? 

Educational Performance?

StudentsFirst just released their state rankings.  You know who came in first?

Louisiana is the top-rated state, according to StudentsFirst. It ranks 49th of 51 on eighth grade reading scores and 47th of 51 on eighth grade math scores.

Washington, DC (dead last on NEAP scores) comes in 4th in the nation for educational performance as rated by StudentsFirst.  Now it is possible that these scores predict improvement in these states and that this is the result of bad past policy.

It's also interesting to see some of the categories.  Look at page 77 of the methodology report and see that they lower the rating of school system that restrict class sizes (bigger classes = better) and, even more interestingly, rate a state more poorly for having a defined benefit pension plan.  How does a defined benefit pension plan and the absence of class limits IMPROVE education.  I could imagine these being orthogonal to educational outcomes and thus argued to be options for fiscal improvement.  But it is hard to argue that they worsen student outcomes.

The Ddulite Bifurcation

From inappropriate aggregation to silly juxtapositions.

For the original definition of Ddulite check the link. For now it's sufficient to say we're talking about people (particularly journalists) who have an emotional, gee-whiz reaction to technology without really thinking seriously about the functionality.

Ddulite journalists can be spotted by a few defining characteristics: a remarkable ability to be impressed by the unimpressive; a focus on shiny, sexy toys; a tendency to report on technologies that really aren't that close as being just around the corner; a recurring amnesia about the slow development of similar technologies; general obliviousness to questions about implementation and demand; and what we might call the ddulite bifurcation.

The typical bifurcation consists of two applications of a new technology, one application mundane but realistic, the other impressive but so wildly ambitious that it may not even be theoretically possible with the technology being discussed.

I was going to make up an absurd example here but now that I think about it, I'm not sure I could do better than this actual story from Planet Money. The subject is 3-D printers and it's worth listening to.The Planet Money people are good, solid reporters and they do a reasonable job putting things in economic context, even bringing in Tyler Cowen to shoot down some of the more extravagant this-is-the-future claims.

But you can count on any story like this to have at least a few ddulite moments and you can certainly find them here, including this classic bifurcation. First we get this claim from a CEO named Pete Weijmarshausen:

Now, I think in a few years, we can print clothing, and then you can have clothing without sizes, but you have the size that fits you.

(Note the qualifiers here: "I think"; "in a few years.")

This is followed a few lines later by analyst Terry Wohlers saying:

WOHLERS: You lose a finger, you print out a new one.
CHACE: Yeah, like, actual body parts, printing out new fingers using your cells.
WOHLERS: Bones and bladders and eventually kidneys and so forth.

(glad he put the "eventually" qualifier with kidneys)

At the risk of belaboring the obvious and working under the assumption that most of you reading this know waaaay more about regenerative medicine and therapeutic cloning than I do, the day when we can easily grow new limbs is probably not just around the corner. Important fundamental research is being done and it's reasonable to talk about being able to do this someday but it could be a long way off. As for 3D printing approaches, we seem to be at the appears to be theoretically possible stage where we can work with masses of tissue rather than just a few cells by creating synthetic vascular systems.

This is exciting research but it's the sort of thing that's probably years away if it ever proves viable. Like most reporting about nanotech, the story mixes the ongoing with the theoretically possible in a way that obscures the huge gap between the two.

Professors and adjuncts -- inappropriate aggregation watch at Forbes

As a follow-up to Joseph's last post on this much-maligned piece from Susan Adams at Forbes, I'd like to add yet another complaint based on this bit from Adams' post:

As for compensation, according to the Bureau of Labor Statistics, the median salary for professors is $62,000, not a huge amount of money but enough to live on, especially in a university town.

Another boon for professors: Universities are expected to add 305,700 adjunct and tenure-track professors by 2020, according to the BLS. All of those attributes land university professor in the number one slot on Careercast.com‘s list of the least stressful jobs of 2013. 

As I've mentioned before, adjuncts have no job security and work very hard for little more than kind words and Pez (the median being a lot less than 62K). By jumping from a statement about professors to one about professors and adjuncts, Adams is using a variation on the rhetorical deception I call the cigarettes and cocaine argument.

"We just don't have the money for you to keep smoking. Do you realize that between your smoking and my cocaine habit we're spending more than two thousand dollars a week? You're just going to have to give up cigarettes."

The gold standard of the C and C argument is the ever popular case for Social Security reform that goes like this:

I. We have to do something about SS

II. The combined costs of SS and Medicare is projected to be more than a gazillion dollars in the red by twenty-whatever

III. That's why we need to cut/privatize/phase out SS

Of course this is just one of many problems in Adams' piece (as Joseph said, James Joyner does a great job addressing the major flaws here), but inappropriate aggregation is an embarrassingly large part of the public discourse and it supports any number of bad arguments and ill-conceived policies despite being relatively easy to spot and correct.

Professor is low stress?

Wow.  Just wow.  James Joyner has a great response to this article on how low stress University Professor is as a job.  I especially liked:

The other thing most of the least stressful jobs have in common: At the end of the day, people in these professions can leave their work behind, and their hours tend to be the traditional nine to five.
 

Really?  That is a rare state of affairs for a professor and is mostly experienced by senior faculty on the verge of retirement.  But developing classes, doing research, and writing grants is not a time limited activity that can be trivially executed in a nine to five sort of way.  Living withy constant uncertainty about funding and whether you will still have a job next year (dependent on successful of grant applications) is also not a low stress lifestyle. 

I don't want to say that there are no professors like this . . . but there are members of any profession who manage to get a very cushy position.  But it seems very unlike the lifestyle I see for the median member of the profession. 

You want flying cars? I'll give you flying cars

As mentioned before, I'm a fan of David Graeber's recent essay, "Of Flying Cars and the Declining Rate of Profit" but I really dislike the title.  Flying cars have become the go-to cliche when discussing underperforming technological progress. On top of that, they have a slightly goofy quality and often come with the at least the implication that no one serious ever actually believed this stuff.

The last part is especially unfortunate because for most of the Twentieth Century, personal aviation was seen as something very close that was going to be very big. Exactly which technology (flying cars, personal planes, and, in the post-war era, jet-packs) would come to dominate was an open question, but serious people believed that flying would become very much like driving for things like commuting and they were willing to back up that belief with money and research.

I n 1933 the U. S. government spent half a million dollars to produce a ‘poor man’s airplane through the efforts of Eugene Vidal, promising a 2-3 seat, all metal aircraft costing $700 (the approximate price of a nice car and considerably less than any aircraft). While this effort was not embraced by the aircraft manufacturers of the time and portrayed as “an all mental aircraft”, the idea was enthusiastically greeted by the public. A direct result of this research was the Erco Ercoupe, which achieved new levels of ease of use, along with a spin-proof, safe stalling, smallfield capable, inexpensive aircraft. T.P. Wright, the Administrator of Civil Aeronautics, wrote an extensive review of NACA small aircraft efforts to “meet the needs of the family”. “When the market for all other types of planes is grouped it is apparent that what may be termed a really large industry, and one having an important effect on national economy, will not be provided. Of course the market for military aircraft will for a long time represent possibly the most important field in aircraft development and manufacture. However, even considering this with the others it can readily be seen that, developed to an adequate extent, the personal aircraft can easily become the most important factor in the aircraft industry. Used both for business and pleasure it is here only that an almost limitless potential market is available.”

Vidal was so committed that he even used his young son to demonstrate (at least briefly) how safe and easy flying these aircraft could be.

Gore Vidal, born Eugene Luther Gore Vidal Jr. on Oct. 3, 1925, in West Point, N.Y., was the only child of First Lieutenant Eugene Luther Vidal and Nina Gore, a socialite. His father was the first aeronautics instructor of the U.S. Military Academy and later the director of the Commerce Department's Bureau of Air Commerce during the Roosevelt Administration. Vidal's father had so much faith in the Hammond flivver-type plane that he sent 10-year-old Gore aloft to fly it. Vidal is pictured at the controls before takeoff. 

The flying car starts looking a bit less goofy in this context. Personal aircraft were soon supposed to be common. Neighborhoods would have their own airstrips. The idea of an airplane that was easily transportable and could double as a family automobile had obvious appeal.

By the Forties, these ideas had even reached the prototype stage

Taylor's design of a roadable aircraft dates back to 1946 [first flight 1949]. During a trip to Delaware, he met inventor Robert E. Fulton, Jr., who had designed an earlier roadable airplane, the Airphibian. Taylor recognized that the detachable wings of Fulton’s design would be better replaced by folding wings. His prototype Aerocar utilized folding wings that allowed the road vehicle to be converted into flight mode in five minutes by one person. When the rear license plate was flipped up, the operator could connect the propeller shaft and attach a pusher propeller. The same engine drives the front wheels through a three-speed manual transmission. When operated as an aircraft, the road transmission is simply left in neutral (though backing up during taxiing is possible by using reverse gear.) On the road, the wings and tail unit were designed to be towed behind the vehicle. Aerocars can drive up to 60 miles per hour and have a top airspeed of 110 miles per hour.

Mid-century Americans had every reason to have high expectations for this type technology. The past fifty years had seen far cruder prototypes of technology such as the car, airplane and helicopter develop into impressive and commercially viable machines. With the Depression and the war out of the way, there was every reason to believe that the turn-around time from early working model to full production would only get faster. If they could build one jet pack today, surely they could have the bugs worked out in a year or two.




That's a Bell Rocket Belt, in case you're curious.

We could argue about exactly why personal aviation never grew beyond the small niche it has occupied for the past few decades, but there's no question that a time traveler from fifty years ago would be surprised at our lack of progress in this area.

Nor do we have a lot of progress to report in the rest of transportation. I'm still not sure how to explain why we actually regressed in terms of transatlantic travel speeds from what we were doing thirty years ago.

Of outhouses and automobiles

Both Megan McArdle and Matt Yglesias have interesting posts about how important network effects can be to the adoption of new pieces of technology (Megan is talking about indoor plumbing, Matt about automobiles).  An except: 

Saying that people are choosing the a cell phone over an outhouse is not the same as saying they’re choosing a cell phone over an indoor toilet. Maybe that’s the choice they’d make, if they had it—I don’t know! But as Kelly’s own account acknowledges, they don’t actually have that choice, and certainly not at anything like the same cost.

Indoor plumbing requires either electricity to pump the water, and a nearby well to pump it from, or a connection to a public system with enough pressure to force the water high enough to flush your toilet. That’s a lot of power, not a trickle charge off of a small solar cell; I believe my great grandparents used a gasoline generator when they installed indoor plumbing in the mid-thirties. Gasoline generators are fairly expensive, as is the gasoline to run them, and I gather that they were only able to do it because their newly married son (my grandfather) saved up to help pay the installation cost, and then paid them rent that covered the cost of the fuel. Most farmers, I am told, waited until rural electrification brought them grid power.
 

Mark also pointed out just how important these elements of infrastructure were in transforming American society.  It's humbling to think about just how much effort was required to actually do all of these things (and concerning that infrastructure moves much slower today). 

However, I am hoping that the shift to an information based economy will have other benefits.  In some sense, there is a possibility that information, stored as pixels, could be something of real value (think of books or television programs) yet require very little resources to create.  In that sense maybe we could end up being happier (overall) while using less resources.

That being said, I have also used an outhouse and have absolutely no interest in giving up my indoor plumbing.  I am not even all that happy camping, unless there is a rest area in the middle of the campground with flush toilets (essential) and showers (highly desierable). 

"Back when 50 miles was a long way"

Over at Statistical Modeling, Causal Inference, and Social Science, Phil picks up on the then-and-now theme and points us to this post from Michael Graham Richard with historic maps showing travel time from New York City in 1800, 1830, 1857, and 1930.

The maps are worth spending some time with (as is the comment section on Phil's post). They're also a nice segue to this observation from David Graeber's excellent essay:

Toffler’s use of acceleration was particularly unfortunate. For most of human history, the top speed at which human beings could travel had been around 25 miles per hour. By 1900 it had increased to 100 miles per hour, and for the next seventy years it did seem to be increasing exponentially. By the time Toffler was writing, in 1970, the record for the fastest speed at which any human had traveled stood at roughly 25,000 mph, achieved by the crew of Apollo 10 in 1969, just one year before. At such an exponential rate, it must have seemed reasonable to assume that within a matter of decades, humanity would be exploring other solar systems. 

Since 1970, no further increase has occurred. The record for the fastest a human has ever traveled remains with the crew of Apollo 10. True, the commercial airliner Concorde, which first flew in 1969, reached a maximum speed of 1,400 mph. And the Soviet Tupolev Tu-144, which flew first, reached an even faster speed of 1,553 mph. But those speeds not only have failed to increase; they have decreased since the Tupolev Tu-144 was cancelled and the Concorde was abandoned.

Miles instead of years

Phil has a great follow-up to Mark's when 50 years was a long time ago post.

Enjoy.

P.S. I was personally struck by just how isolated the Puget Sound area really was in 1857; 6 weeks is an amazingly long trip to still be in the same country.

P.P.S. I should look at the scheduled posts before saying anything.  Suffice it to say, a post with a lot more actual thinking is forthcoming and, having run across it in the archives, it is worth the wait

Megan McArdle's collectivist libertarianism

McArdle has come in, quite rightly, for a great deal of criticism following her recent post suggesting:

I’d also like us to encourage people to gang rush shooters, rather than following their instincts to hide; if we drilled it into young people that the correct thing to do is for everyone to instantly run at the guy with the gun, these sorts of mass shootings would be less deadly, because even a guy with a very powerful weapon can be brought down by 8-12 unarmed bodies piling on him at once.

In terms of tactics, simply rushing a gunman whose weapons may be fully automatic is an extraordinarily bad idea, truly a last resort. It looks good in the movies but in real life, every aspect of the maneuver -- range, position, response time -- plays to the shooter's advantage. This is pretty much the situation that assault weapons were designed for.

In terms of implementation, it is arguably even less practical. As an old history professor of mine (who happened to be ex-military) explained, when someone shoots at you, the overwhelming instinct is to run away which is why so much military culture is designed to condition soldiers to reflexively follow orders (and why officers sometimes point their sidearms at their own troops).

Jonathan Chait pointed out that even in the familiar and controlled setting of a football practice, it takes considerable training to get kids to rush toward large, threatening opponents without hesitating or flinching. The idea of getting typical elementary school children to instinctively swarm an armed gunman is so absurd that Chait concluded:

Unless I am missing a very subtle parody of libertarianism, McArdle’s plan to teach children to launch banzai charges against mass murderers is the single worst solution to any problem I have ever seen offered in a major publication.

That's the one part I disagree with, not about it being the worst solution but about it being libertarian. McArdle is suggesting that we institute what can only be a massive government program to indoctrinate kids to put aside personal choice and individual initiative and instead automatically take collective action to serve the interests of the group. I honestly can't think of a recent proposal more at odds with libertarian principles.

This last point has no real significance in the gun control debate. McArdle's idea was next to impossible to implement and was unlikely to work even if you could get it in place. The fact that it contradicted her stated core values has no bearing on the question of guns and safety.

It does matter, however, when we consider the larger and, in the long run more important, question of how to have a discussion (more important than keeping our children safe? Yes. We have to be able to intelligently discuss the problem before we can hope to address it). A great deal of our discourse on almost every major issue is staked out by nominal libertarians like McArdle.

Libertarianism is often treated as the respectable and intellectually coherent branch of conservative thought, particularly when compared with say. social conservatives or nativists, but if you start with the same axioms these groups hold about the validity and interpretation of certain sacred texts or about cultural identity respectively, then most of the positions held by social conservatives and nativists are at least coherent. By comparison, much. perhaps most, of what we hear from leading libertarians like McArdle is completely inconsistent with the defining assumptions of libertarianism.

With a few exceptions, most of the nominal libertarians seem to take a curiously pro-authority stance, particularly when that authority preserves the social order. Even when the authority is governmental, actions that greatly reduce aggregate liberty (the war on drugs, extensions of copyrights and patents), are objected to less strenuously than are policies that arguably increase aggregate liberty such as civil rights laws.

Update: You can see Megan's response to this post in the comment section.

When Patents attack

Kevin Drum has an interesting story about a software patent case:

This hits home for me in two ways. First, the alleged patents date from 1996, and I was personally involved in a project to put scanners on networks starting around 1994. It was cleverly called NetScan, and it eventually failed for a variety of reasons, but by 1996 we had an actual box on the market that allowed you to connect a scanner and program it to send documents to your internal email account. I have no doubt that the patent trolls in this case would argue that the technology we used was subtly different from theirs (we emailed TIFF files, for example, while their patent covers PDFs), but that's almost certainly legalistic nonsense. You connected a scanner to our box, entered a bunch of data identifying users, and then you could scan documents and have them automatically emailed to your desktop. We didn't even bother patenting it because the idea was pretty obvious.

I think that this makes it pretty clear how silly a lot of modern patent law has become.  There is not really any innovation being protected here and, instead, we have a lot of lawyers becoming rich because somebody decided to take out a patent on what people were already doing. 

This makes me extremely skeptical that patents are a direct correlate with innovation, unless you sub-group them very carefully.   And it is highly disturbing to see small businesses (which don't have deep pockets for legal fees) being increasingly targeted by patent lawyers.  There are already a lot of barriers to being a small business.  A catalogue of patents for simple things (like scanning a document to email) would be cumbersome and trying to be compliant with it would be the most onerous set of regulations I can imagine.

Bad grading idea

Via Tyler Cowen:

 I will submit your papers (blind) to external referees as well as myself for assessment, an A grade will be limited to those papers, and only those papers, that are recommended for acceptance or conditional acceptance, a B grade will be assigned to those papers that receive a recommendation of revise and resubmit, and a C grade will be assigned to those papers that are rejected by the external referees and myself.
 

I would be quite annoyed to discover that I was putting in the hours to evalaute a paper only to discover that I was doing a professor's job of grading said paper.  Furthermore, it seems that the editor is also the professor for the course.  I would be reluctant to evaluate a student or peer at tmy institution.  The less distance, the more I would be reluctant to do so.  The professor in queestion is willing to blind the papers for the external reviewers, who can not possibly be as potentially biased as a professor with their own students. 

I am also wondering about the standards of a journal in which revise and resubmit is a B grade.  There cannot be many A's.  I have (once) had a paper accepted without revisions but it was definitely not the first time it was ever sent to a journal.  The idea that a paper done in a single semester course (in parallel with other classes) would be a paper so high quality that it was accepted without revisions less than 4 months of work would be incredible in Epidemiology.