Survivor bias

There has been some discussion about meony from parents leading to a lower GPA from both Andrew Gelman and I.  One of the comments at Andrew Gelman's site got me thinking:

” The higher graduation rate of students whose parents paid their way is not surprising, she said, since many students leave college for financial reasons. (…)
Oddly, a lot of the parents who contributed the most money didn’t get the best returns on their investment (…) Their students were more likely to stay and graduate, but their G.P.A.’s were mediocre at best, and some I didn’t see study even once.”
 

What is the actual target of inference here?   Is it GPA or is it graduation? 

When I was in the corporate world I never was asked for my transcripts (my degree, all of the time but my transcripts never).  Having 2 years of college and then dropping out leads to worse life outcomes than having a degree, so far as I can tell. 

Or put it another way, what would you prefer:

1. A child who got high marks but did not complete their program?
2. A child who got low marks but earned a degree?

Insofar as university education is a credentaling and signaling system, the second piece would be far better. 

But even more interesting, the authors comments support my intuition precisely -- parental funding keeps marginal students in school.  From a causal perspective this is way more interesting than the headline effect of giving dropping grades and is way more intuitive as well. 

 

Health Care spending

I often worry that comparison between the United States and Canada understate the room we have to improve health care costs.  But Matt Yglesias points to a health care chart showing spending per capita by the government in the United States and Canada:

This is the chart that I think ought to dominate the conversation about public sector health care spending in the United States and yet is curiously ignored. The data show government health care spending per capita in the United States and Canada. The United States spends more. And that's not more per person who gets government health insurance, it's more per resident. And yet Canada covers all its citizens and we don't. That should be considered shocking stuff, and yet I rarely hear it mentioned.

Even odder is that the most recent time I heard it mentioned was Valerie Ramey talking at the American Economics Association conference in San Diego and her conclusion was that this showed U.S. health care needs free market reforms. The more straightforward interpretation, I would think, is that the U.S. needs to make its system more like Canada's. It's important to note that the example here is Canada. Not some radically different society. Not some far-off distant land. And the gap is actually growing.

In 2010, the Canadian government was spending roughly ~$3,000 per capita and the US government was spending ~$4,000 per capita.  Not per beneficiary, but per capita!! 

Now Canada is not some sort of dystopia without private medicine.  Emergency services are fully covered but it was common when I was last there to go to a private clinic for health care like an X-ray to avoid the queue in the public options.  But this is still catastrophic coverage for all citizens, which is an impressive feat. 

At the end of the post, Matt talks a little about the less innovation counter-argument.  Closely related is the MD shortage argument.  Now there are two responses.  One, is to note (as Matt does) that we want health care costs to go down in the United States and that this will have bad effects as well as good effects. 

But the second is more compelling.  We could just invest money in medical research and in awarding prizes for drug discovery.  It would make innovation costs transparent and separate it out from rent-seeking and administrative costs (which would be lower in the single-payer insurance or out of pocket expenses world). 

As for educating MDs, we can steal a leaf from Canada and subsidize education a wee bit more and lower wages will still have no trouble attracting people to a high status and still relatively high pay profession.  No solution is perfect, but why is this one not being debated?

More reflections on a study (EDITED)

Okay, before I get to the meat of this post, this quote by Andrew Gelman is dynamite:

. . . I’m generally suspicious of arguments in which the rebound is bigger than the main effect.
 

How many "counter-intuitive" studies would survive this kind of skepticism.  Not that a rebound effect can't be larger, but like many unlikely things it requires a higher level of proof.

The context is an education study which suggests that the more parents pay for education the lower the grades of the student will be.   The authors apparently tried to control for a lot of possible confounders (like SAT scores) but the whole process ends up looking like "what not to do in regression analysis". 

There is an intermediate variable (problem), a restriction of range problem (extrapolating parental support out to values that exceed annual income), and an issue with differential drop-out that does not seem to be addressed.  All of these points are present in Andrew's nice write up. 

What I want to focus on is the sharp counter-factual.  I am not always a fan of counter-factual reasoning, but I think that it would provide a ton of clarity in this case.  The real claim is that if you decreased exposure X (parental support) then you would increase outcome Y (GPA).  The direct causal model would suggest that the fastest way to improve student grades would be to make your contributions zero.  But, the last time I looked, Pell grants require a non-zero parental contribution in most cases (it is a little hard to tell precisely what the thresholds are but they definitely are not zero for most students).  So clearly this is a floor on parental contributions (and if it was the only source of contributions the effect would become the richer the parents the worse the grades of the student). 

So maybe, to have a realistic counter-factual, the exposure should be dollars of support above the minimum expected contribution? 

So, really what we have to be talking about the the effect of a marginal dollar separate from the (non-linear) scale of what the parents are required to pay.  But, even there, the direction is unclear.  Imagine that your not especially inspired child gets admission to Stanford but they are struggling with the material.  Do you insist, on principle, that they get a job or do you pay more so that they have a better chance to be a "C average" Stanford graduate (which is much better than a Stanford drop-out). So the causal direction is actually unclear. 

But if the idea is that giving more resources to students decreases performance then there are a lot of experiments we could try.  For example, we could decrease wages (for everyone including upper managment) and see if performance goes up.  Or we could randomize students to improved levels of support.  Better yet, we could look at experiments that have already been done:

 

We examine the impacts of a private need-based college financial aid program distributing grants at random among first-year Pell Grant recipients at thirteen public Wisconsin universities. The Wisconsin Scholars Grant of $3,500 per year required full-time attendance. Estimates based on four cohorts of students suggest that offering the grant increased completion of a full-time credit load and rates of re-enrollment for a second year of college. An increase of $1,000 in total financial aid received during a student’s first year of college was associated with a 2.8 to 4.1 percentage point increase in rates of enrollment for the second year.
 

 So not only is the main effect in the opposite direction (at least in terms of retention) but it has precisely the impact on a GPA analysis that Andrew expects: students are more likely to leave with lower levels of support.  Do we think that leaving school is completely independent of performance (that there is no GPA difference between the drop-outs and those who persist)?  Or is parental support different, in some magic way, than government grant support?  People are more careful stewards of government money than they are of money from their close community (and think about what this would mean for charity versus government welfare programs, if true)?

I agree that the current form of this study is impossible to interpret. 

[EDIT: Talking with Mark, it is clear that I was unclear on one point above.  The experiments show money from a specific source (i.e. government funding) go in a specific direction but don't at all address whether money from parents has a similar causal effect (Mark is promising to talk about this in a post himself).  The issues of selection, intermediate variables, and experimental evidence from other sources are all important, but without re-analyzing the data it is impossible to prove the directionality of the bias.  As an epidemiologist I am trained to speculate on bias direction/strength but I recognize that is all I am doing.  ]

One more cognitive dissonance post

Andrew Gelman has a good, skeptical response to some questionable claims from Herbalife, but there's one point where we're in disagreement, not so much as to the conclusion as to the reasoning behind it.

Gelman says:

Amusingly, one of Herbalife’s points is “Fact: Majority of Former Distributors Would Recommend Herbalife to Friends and Family.” But that’s exactly what you’d expect of a still-active pyramid scheme, no? Existing members want new people below them on the pyramid. I’m not saying this means it is a pyramid scheme, but it doesn’t seem like evidence against the hypothesis!

Perhaps I'm misreading this but I'd assume that former distributors no longer have a direct interest in the company. If this is true, does this mean we can take former distributors as impartial judges? Not by a long shot.

This is where we segue back to a recent thread, cognitive dissonance and the psychology of marketing. Companies like Amway and Herbalife are textbook examples of marketing psych (literally for Amway, in two different chapters, no less). And it's important to remember that this relationship goes both ways. While the psychologists writing these texts study these companies, many of the executive in these companies have read these books and taken these classes and they've thought seriously about how best to apply these principles.

[My background in this field is spotty so I would highly recommend that you pick up a copy of Cialdini's Influence (either regular or textbook version) or some other good text on the subject and make sure I'm getting this right... ]

There have been a number of studies that show that when you convince people to believe something based on one reason, they have a tendency to come up with additional reasons to support that belief and that these reasons do not go away just because the original reason is removed. This effect is even stronger when the belief is stated publicly, particularly to friends and family or in writing (Amway training makes a big deal about getting things in writing).

The former Herbalife distributors are another one of those cases where what happened was exactly what the textbooks said would happen: people who had sold a line of products to friends and family in the past now tend to hold the reassuring belief that those products were good. This doesn't prove that they weren't good and it certainly doesn't say anything one way or the other about Herbalife being a pyramid; it simply serves as another reminder that things often happen the way your professor said they would.

Alexandria Word Searches -- a new kind of puzzle for the weekend

I've got a post on a different kind of word search puzzle at You Do the Math. Mainly pitched toward teachers, but hopefully still fun for the general audience. Here's a Shakespeare-themed example with, at last count, eighteen Bard-friendly answers.

There are more puzzles (in larger formats) at the original post.

"Edward Tufte Wants You to See Better"

I was looking for an interview for a post on the way we cover health issues, when I came across this interview with Edward Tufte. I haven't had a chance to check it out yet so this isn't really a recommendation but, given the depth of my to do list, I decided it would be better to pass it along now.

Just another (incredibly bizarre) data point

I dislike posts and op-eds that use the latest big news story as an excuse to mount a favorite hobby horse, but I can't pass this one up.

One of the recurring themes here has been the decline in journalistic standards, particularly regarding accuracy and fact-checking. This brings us to the incredibly strange case of Manti Te'o and the imaginary dead girlfriend. It was one of the year's most widely covered stories. Everybody from Sports Illustrated to CBS to the New York Times  ("He has personified hope after more than a decade of mediocrity. He has lived the university’s core values at a place where that matters, said Athletic Director Jack Swarbrick.") had carried moving accounts of Te'o's story, but it wasn't until a few days ago that the sports site Deadspin actually dug into the details.

Notre Dame's Manti Te'o, the stories said, played this season under a terrible burden. A Mormon linebacker who led his Catholic school's football program back to glory, Te'o was whipsawed between personal tragedies along the way. In the span of six hours in September, as Sports Illustrated told it, Te'o learned first of the death of his grandmother, Annette Santiago, and then of the death of his girlfriend, Lennay Kekua.

Kekua, 22 years old, had been in a serious car accident in California, and then had been diagnosed with leukemia. SI's Pete Thamel described how Te'o would phone her in her hospital room and stay on the line with her as he slept through the night. "Her relatives told him that at her lowest points, as she fought to emerge from a coma, her breathing rate would increase at the sound of his voice," Thamel wrote.

Upon receiving the news of the two deaths, Te'o went out and led the Fighting Irish to a 20-3 upset of Michigan State, racking up 12 tackles. It was heartbreaking and inspirational. Te'o would appear on ESPN's College GameDay to talk about the letters Kekua had written him during her illness. He would send a heartfelt letter to the parents of a sick child, discussing his experience with disease and grief. The South Bend Tribune wrote an article describing the young couple's fairytale meeting—she, a Stanford student; he, a Notre Dame star—after a football game outside Palo Alto.

Did you enjoy the uplifiting story, the tale of a man who responded to adversity by becoming one of the top players of the game? If so, stop reading. Manti Te'o did lose his grandmother this past fall. Annette Santiago died on Sept. 11, 2012, at the age of 72, according to Social Security Administration records in Nexis. But there is no SSA record there of the death of Lennay Marie Kekua, that day or any other. Her passing, recounted so many times in the national media, produces no obituary or funeral announcement in Nexis, and no mention in the Stanford student newspaper.

Nor is there any report of a severe auto accident involving a Lennay Kekua. Background checks turn up nothing. The Stanford registrar's office has no record that a Lennay Kekua ever enrolled. There is no record of her birth in the news. Outside of a few Twitter and Instagram accounts, there's no online evidence that Lennay Kekua ever existed.

The photographs identified as Kekua—in online tributes and on TV news reports—are pictures from the social-media accounts of a 22-year-old California woman who is not named Lennay Kekua. She is not a Stanford graduate; she has not been in a severe car accident; and she does not have leukemia. And she has never met Manti Te'o.

It has since come out that some journalists had checked some facts and noticed something was wrong but not wrong enough to keep them from running this incredibly dramatic but completely untrue story.

For a reporter looking for a touching human interest story, this was too good to be true and many of the nation's biggest and (sadly) best news organizations ran it without bothering to determine if it was true. That's  troubling.

But unfortunately no longer all that surprising.                                                                                              

Ray Fisman has a new book you might want to check out

Though we've disagreed strongly in the past, Ray Fisman is a smart guy with some insightful things to say about a problem Joseph and I have been thinking a lot about, how interests go out of alignment and how we can engineer big, complex organizations to keep that from happening.

Fisman has a new book out on the subject: “The Org: The Underlying Logic of the Office.” Here's a quote from an enthusiastic review in the New York Times.

This suggests a good rule of thumb to determine when a private company will outperform the public sector: if the task is clear-cut and it’s possible to define concrete goals and reward those who meet them, the private sector will probably do better. “If I can write a perfect contract in which I pay for a concrete observable outcome, can rule out cream-skimming and can ensure the measure is not gamed, there is no reason that the private sector can’t do it better,” Professor Fisman said.

Safety Nets and Canada

Dean Dad has an interesting observation:

I was reminded of that a few days ago, in a discussion with a Canadian colleague.  We have similar senses of humor, so we got to talking about The Kids In The Hall, SCTV, and national styles of humor.  (For my money, “Brain Candy” is a neglected classic of dark, dark, dark comedy.)  She offered the theory that Canada punches above its weight culturally because its social safety net -- health care most conspicuously -- makes it possible for people to take chances on creative careers.  As a result, they get Holly Cole, and we’re left with Adam Sandler. 

That was then expanded on in the comments

While I disagree with the specific point about Canada punching above its weight culturally (quick name a great Canadian film that's not "Strange Brew"), I do think that a robust safety net does make entrepreneurial risk taking more likely because people can afford to take the risk of starting a business without having to worry about losing health insurance or other benefits.
I used to have a state government job where this dynamic was apparent: the secretaries in the agency were fairly low paid, but had very good benefits. 3/4 of the secretaries in my officer were married to husbands who had their own small contracting or (vaguely) construction related business. They made much more than their wives made, but had no independent health benefits of their own  

I think that this is a neglected conversation.  The ability to take risks is not just driven by rewards but also by the costs of failure.  If you make the rewards extreme and failure punishing then you create incentives for cheating and "doing anything to win".

This effect shows up in a number of areas -- imagine you are a high school teacher diagnosed with a major illness.  In the real world, COBRA is unaffordable and unemployment is over eight percent.  You are teaching less well due to health issues.  One can see a lot of pressure to find a way -- any way -- to keep test scores above the retention threshold.

You can also see this with small businesses.  The reforms of bankruptcy law (making it harder to go bankrupt) and the cost of health care for those without insurance makes starting up a small firm really risky.  It makes a lot more sense to stay in your sub-optimal office job with the result that you have less innovation and dynamism in the economy.

These effects are just as predictable as free markets are and it can make a lot of sense to invest in ways to pool or mitigate the risk associated with being an innovator.

Stop me if you've heard this one before

Michael Shermer has an interesting post over at Scientific American, but there's one paragraph I'm not too happy about.

Cognitive dissonance may also be at work in the compartmentalization of beliefs. In the 2010 article “When in Doubt, Shout!” in Psychological Science, Northwestern University researchers David Gal and Derek Rucker found that when subjects' closely held beliefs were shaken, they “engaged in more advocacy of their beliefs ... than did people whose confidence was not undermined.” Further, they concluded that enthusiastic evangelists of a belief may in fact be “boiling over with doubt,” and thus their persistent proselytizing may be a signal that the belief warrants skepticism.

I'm way out of my field on this one. My knowledge of psychology is limited to an undergrad intro course and a copy of Cialdini's Influence, but I'm pretty sure that researchers have been finding this sort of thing since the mid-Fifties when Leon Festinger wrote about doomsday cultists proselytizing more the day after the world failed to end and confirming this and related aspects of cognitive dissonance ever since in a large group of studies. (Discover did a better job with the context, though that is comparing a paragraph to a whole post.)

I don't mean to suggest Gal and Rucker were not doing important, original work. I'm sure they were. My beef here is with Shermer and the lack of context. A reader coming on this paragraph cold would be left with the impression that this was a new idea rather than the latest brick in a decades long wall.

I can understand the appeal of the cutting edge. The new stuff is sexier. It gets people's attention. The trouble is, those cutting edge studies often collapse under scrutiny. Some can't be replicated. Others prove to be not that important.

Confirmation, on the other hand, is not sexy. It doesn't drive traffic. It's harder to fit into a paragraph. In a way, though, it's more interesting because it has a high likelihood of being true and fills in the gaps in big, important questions. The interaction between the ideas is usually the interesting part.

Of course, this may be awfully picky given that we're talking about a single paragraph, but this is a recurring issue. New developments are frequently reported in a vacuum, and the result is often a badly misled reader. In these situations a few lines of context can go a long way.

Misleading graphics

This infographic has been coming under criticism from the usual suspects.  The group that got hit the worst was the married couple with two children making $650,000 per year.  According to the census bureau, in 2009 the median income for a household was $49,777.  So there is a tax increase of 3.3% on couples making 13 times the median US household income. 

Needless to say, this really doesn't reflect the likely impact of these tax law changes on the typical American (it is just to easy to ask hard questions about the representativeness of a single mother who makes $260K/year). 

[note -- label typo corrected]

More thoughts on Education

A couple more thoughts on the whole education reform movement, both hoisted from comments.

First, there was a comment by Stuart Buck:

As a matter of basic social science, what should concern one is not the absolute level of a state's performance now but the counterfactual (what would its performance otherwise be).
 

This is absolutely correct.  However, what we are really missing is a time frame for improvement as well as an expected magnitude of improvement.  So if we look at 1999, the top rated state in StudentsFirst (La) had a Grade 8 reading score of 252 (compared to an average of 261).  In 2011 the score was 255 (improved 3 points) versus a national average of 264 (which also improved three points).  Between 2009 and 2011 both La and the national average also improved by the same amount.  DC is even more interesting.  In 2007 (when Rhee began her reforms) Grade 8 reading was 241 (versus 261 nationwide).  In 2011 it was 242 (versus 264 nationwide). 


So we would have to explain the unexpected drop in performance we expected in Louisiana that simply did not happen or the reason by DC lagged even further 4 years into a reform program.  Even longitudinally, first in the nation seems odd given a lower baseline (thus more room for improvement due to lower hanging fruit and maybe even regression to the mean) and an absolutely dead standard increase. 

Second, at lawyers, guns and money a commenter said:

That being said, using it for evaluative purposes is misguided and unfair to educators. I proctor the test, and I see a large number of students who don’t take the test seriously at all. They just click through to get it over with. Our student population has taken the test in the grips of a horrible flu outbreak. Those kids who were actually in school at the time were sick, getting sick, or struggling to get over being sick. When you have to spray down the computers with Lysol after every class comes through, you really have to question the validity of the results obtained. Technical difficulties that require restarting the computer and/or test can also have a suppressive effect on students’ scores.
 

As the Tech coordinator in a school, this seems to be a reasonable position to make such an evaluation.  That raises the question of "high stakes for whom"?  I am actually a fan of looking at SAT scores.  Why?  Because not only is the test well respected but the test makers have a financial incentive to make sure the test does what they say it does (so it can continue as a national standard).  The students have an incentive to do well on this test because high scores open doors for them.  So when a teacher is evaluated on SAT performance, I am pretty comfortable saying that the other actors are likely to have aligned incentives on giving an unbiased estimate.

Finally, the thing that really seems to be mixed up in the Rhee report is the difference between efficiency (cost savings) and quality (performance).  By analogy, consider military pensions.  They exist, in large part, so that we can retain top performers in the armed forces.  If anything the defined benefit pensions improves quality by keeping soldier with 15 years of valuable experience in the military.  The problem with pensions only arises if the military gets bad at weeding out incompetent performers (which, so far as I can tell, is not currently a major problem).  It is good to keep experienced people around while they are still effective but it is expensive.  So the empirical question is does it cost more than it is worth?

The same issue arises with the class size metric.  I have been in large and small classes with an excellent teacher.  I learned a lot more in the small class because the teacher could focus more attention on each student.  Is it better to have large classes (like StudentsFirst claims)?  Well, only if you have identified top performers and can assure yourself that you are compensating for class quality with teacher quality.  This is a hard claim to support.  On the other hand, almost no luxury is as expensive as small classes.  Notice how universities have reacted to this pressure by putting hundreds of students into a single classroom.  So is the cost worth the improvement in quality is a legitimate question. 

So the issues here are twofold.  One, the data on performance do not seem to map easily onto the counter-intuitive rankings of StudentsFirst.  Two, the type of high stakes test that seems to be a key feature of the education reform movement has some work to do in properly aligning incentives. 

 

Hat tip to Yglesias

Mark and I have been hard on Matt Yglesias lately.  But this post was a really clever idea.  He took Kevin Drum's idea that lead was responsible for a rise in violent crime and asked "if this explanation is correct then what else must be true?".  Since impulse control is also linked to high school graduation he plotted high school graduation (which has been unexpectely rising) and sees the same pattern as with violent crime. 

Now nothing rules out a confounder that affects both of these outcomes.  I worry about simple explanations for complex phenomena.  But it was a clever idea to try and falsify the hypothesis by looking at things that should be related to violent crime.  And, if there was going to be a candidate for such a broad level of toxicity, a substance implicated in brain damage going back to classical Rome isn't a bad choice. 

Felix Salmon on personal finance

Felix Salmon has beaten me to the punch here but I do think that this statement needs to be properly understood for what it means:

It surely comforts modern parents who have spent fortunes educating their children to know that these children are spending money on pork belly and not, for instance, cocaine. But what solace can it offer to realize that $300 a week put into an S. & P. 500 Index fund over the past five years would have provided an annual rate of return of 10.34 percent and grown to $100,354 today? Even saving $300 a week at a 6 percent rate of return would have yielded about $91,000, Mark X. Chemtob, a financial adviser at Ameriprise, said, adding that in both cases, the sums would qualify for a down payment on a starter apartment in New York.

So if a person invested for five years, and got a retern of 10.34 percent they would have a lot of money.  So have happened 5 years ago (2007)?  Here is wikipedia:

The Dow Jones Industrial Average, Nasdaq Composite and S&P 500 all experienced declines of greater than 20% from their peaks in late 2007. 

So if you had perfect market timing then you could have invested directly after a crash (as opposed to during it) taking advantage of the recent market crash.  Unless, of course, you were the 23 year old in the article who is likely in school and not making $300/week of investable income. 

The other side of this coin is that it is very hard to be 23 years old, just graduated from school, making real money for the first time in your life and not enjoy some of it.  After all, perpetual deferred gratification is never being able to enjoy the rewards of your career.  Nor is it clear that somebody in their first year out of college should be buying a Manhattan apartment (a highly leveraged investment) until they find out if they are going to be successful in New York. 

Nor can you drop the cost to zero.  I would find it hard to eat in New York city for less then $75/week.  Remember, we are taking a city where space is at a premium and everything bought in the city is expensive (including kitchen facilities).  So eating 21 meals at about $3 apiece is actually pretty tough, even if you have good skills for cooking from scratch.  And, even more interesting, the person in this example is taking on extra work to fund her leisure time (as opposed to, for example, debt). 

So I agree -- a very misleading example. 

Technological stagnation bandwagon

Looks like this idea may actually be getting fashionable (from the Economist):

So it may come as a surprise that some in Silicon Valley think the place is stagnant, and that the rate of innovation has been slackening for decades. Peter Thiel, a founder of PayPal, an internet payment company, and the first outside investor in Facebook, a social network, says that innovation in America is “somewhere between dire straits and dead”. Engineers in all sorts of areas share similar feelings of disappointment. And a small but growing group of economists reckon the economic impact of the innovations of today may pale in comparison with those of the past.

Now we just have to figure out what to do about it.