Living with Mistakes the easy way -- larger points

Andrew Gelman fact-checks the New York Times:

For example, David Brooks wrote the following, in a column called “Living with Mistakes”:

The historian Leslie Hannah identified the ten largest American companies in 1912. None of those companies ranked in the top 100 companies by 1990.

Huh? Could that really be? I googled “ten largest american companies 1912″ and found this, from Leslie Hannah:

No big deal: two still in the top 10 rather than zero in the top 100, but Brooks’s general point still holds. As Brooks said, we have to live with mistakes. This is more a comment on how a statistician such as myself will see a number and immediately feel the urge to check it.

...

Again, this is no criticism of Brooks—as a journalist, he’s of course more interested in good stories than in getting the details right (recall the notorious $20 dinner at Red Lobster). That’s ok. Storytelling is his job, numbers are mine.

I appreciate Gelman's sharp eye and I can understand why, being a nice guy, he tends to favor catch-and-release criticism, but I disagree sharply with his conclusions here.

For starters, creative destruction a big part of the story Brooks is telling, both in this column and in his body of work collectively. In this narrative, it's a tough process, but a healthy and fundamentally fair one. Here's the sentence that precedes Gelman's excerpt from Brooks: "Even if you make it to the top, it is very hard to stay there."

We can argue about the validity of this view, but there's no question that Brooks' incorrect statement supported this point while the corrected version undercuts it. There are millions of businesses and if success is truly determined solely by who has the best business model, the best execution and the best timing, a long run in the Fortune 500 (with shifts in markets and changes in management) would be a hell of a feat. The number we actually see would certainly imply something more at work (regulatory capture, anti-competitive practices, etc.).

So yes, I would call this a big deal with respect to the story Brooks is telling.

Perhaps more importantly, I have a problem with the distinction drawn here between statisticians and the rest of the world. It's true that the ability to sniff out suspect numbers and questionable findings is an essential part of being a good statistician, but it's also part being a good journalist (and a good engineer and a good accountant and any number of other professions).

As statisticians we need strong mathematical intuition and a heightened sense of how numbers relate to each other, but we don't have any claim to the urge to check the unlikely. David Brooks was simply being a bad journalist when he wrote that passage. It was not because of inability -- Brooks is smart and highly capable -- but because he didn't care enough to get it right. He know there would be no real consequences either for him or his paper if he got it wrong.

And a lack of consequences, my friends, makes living with your mistakes amazingly easy.

"[T]he numbers show that wage inflation is — literally — the least of the problems when it comes to university cost inflation"

The quote comes from Felix Salmon and it's part of an excellent discussion (nicely summarized at Rortybomb). If you're interested in either education or the economy (where student debt is becoming a major factor), you should read all of the posts, but if I had to pick one point it would be this unbelievable projection cited by Malcolm Harris:

And while the proportion of tenure-track teaching faculty has dwindled, the number of managers has skyrocketed in both relative and absolute terms. If current trends continue, the Department of Education estimates that by 2014 there will be more administrators than instructors at American four-year nonprofit colleges.

Also posted at Education and Statistics.

Probably the damnedest thing you'll see this weekend

And you think you have trouble getting out there for your morning jog...

Peak Life Expectancy

I am often skeptical of these claims that we will see the next generation have a shorter life expectancy as these claims require models. These models may be incorrect for a variety of reasons: misspecification, noise, shifting patterns of disease, unexpected technological improvements, and so forth.

But what was fascinating was the map in the article. The places in the United States where life expectancy is dropping are focused mainly in the Southeast. Now that distribution is, itself, interesting as the southeast has long had health issues: think of the classic stroke belt. Furthermore, it is an area of high inequality that has a climate that is very compatible with a sedentary lifestyle.

Contrast this with the California coast (and especially Los Angeles) where life expectancies are actually rising, or even New York city. Could it be that an urban lifestyle is actually life enhancing (both in terms of quantity and quality)?

So perhaps, instead, what we have is an ecological experiment to really try and understand these phenomena.

Glycemic control and diabetes: today's evidence

This paper has the potential to be pretty important:

Intensive glycaemic control for patients with type 2 diabetes: systematic review with meta-analysis and trial sequential analysis of randomised clinical trials

The article is free online but let me quote from the conclusion:

Intensive glycaemic control does not seem to reduce all cause mortality in patients with type 2 diabetes. Data available from randomised clinical trials remain insufficient to prove or refute a relative risk reduction for cardiovascular mortality, non-fatal myocardial infarction, composite microvascular complications, or retinopathy at a magnitude of 10%. Intensive glycaemic control increases the relative risk of severe hypoglycaemia by 30%.

This is actually quite important. It is very difficult for patients to maintain low levels of blood glucose, with consequences in both quality of life and adverse events. Tight glycemic control, for example is the reason that some policy makers have concerns about diabetics driving (due to worries about hypoglycemic attacks).

Given the difficulty of getting diabetics to adhere to tight glycemic control (and concerns about issues like driving), perhaps we should be more cautious in pushing tight control? More interestingly, we should ask why this association seems non-linear, as it is obvious that objectively poor glycemic control is very cardiotoxic.

But this was a very interesting paper for highlighting what we do and not not know.

Things that make you wince science writing edition

I heard a brief discussion of Milgram's six degrees by KCRW's Sara Terry and NYT's Somini Sengupta. It wasn't pretty, but it did get me thinking about science stories that are consistently misreported. Two immediately jumped to mind:

Milgram's Small World

The Butterfly Effect

Now that we've got things started, I'm opening the floor to nominations. What science stories can you count on to make you wince?

Tax Levels

From Don Talyor:

If we adopted 21% of GDP as a future target for balancing the budget, we would be saying government spending will be less while the baby boomers are eligible for Medicare and Social Security than it commonly was when they were paying taxes to support these same programs. This will be very hard. Plans seeking balance at lower levels seem implausible.

I think that this is worth keeping in mind. A 21% of GDP budget target would already be a painful and difficult process to achieve with the headwinds we have already due to population aging. Lower levels require a really novel idea about how to reverse these headwinds.

But the options that might really shift the balance (immigration) seem to be out of favor right now.

Toys for Tots -- reprinted, slightly revised

A good Christmas can do a lot to take the edge off of a bad year both for children and their parents (and a lot of families are having a bad year). It's not too late to pick up a few toys, drop them by the fire station and make some people feel good about themselves during what can be one of the toughest times of the year.

If you're new to the Toys-for-Tots concept, here are the rules I normally use when shopping:

The gifts should be nice enough to sit alone under a tree. The child who gets nothing else should still feel that he or she had a special Christmas. A large stuffed animal, a big metal truck, a large can of Legos with enough pieces to keep up with an active imagination. You can get any of these for around twenty or twenty-five bucks at Target;

Shop smart. The better the deals the more toys can go in your cart;

No batteries. (I'm a strong believer in kid power);*

Speaking of kid power, it's impossible to be sedentary while playing with a basketball;

No toys that need lots of accessories;

For games, you're generally better off going with a classic;

No movie or TV show tie-ins. (This one's kind of a personal quirk and I will make some exceptions like Sesame Street);

Look for something durable. These will have to last;

For smaller children, you really can't beat Fisher Price and PlaySkool. Both companies have mastered the art of coming up with cleverly designed toys that children love and that will stand up to generations of energetic and creative play.

* I'd like to soften this position just bit. It's okay for a toy to use batteries, just not to need them. Fisher Price and PlaySkool have both gotten into the habit of adding lights and sounds to classic toys, but when the batteries die, the toys live on, still powered by the energy of children at play.

The most disgusting image I could come up with in the education reform debate

You can read my thoughts on dismembered puppy math over at Education and Statistics.

Statements that I violently disagree with

From Tyler Cowen via Scott Sumner:

Congratulations to Matt Yglesias on his new gig. He’s arguably the best progressive economist in the blogosphere, which isn’t bad given that he’s not an economist. I said “arguably” because Krugman’s a more talented macroeconomist. But Yglesias can address a much wider variety of policy issues in a very persuasive fashion. So he’s certainly in the top 5. His blog is the best argument for progressive policy that I’ve ever read. (But not quite persuasive enough to convince me.)

Now do not get me wrong: I post a lot about Matt Yglesias because I think that he is a fine thinker and has some really nice points to make. But there is now way he is competitive to be the top progressive economist in the blogosphere. I can't claim to be an expert but, off the top of my head, I have have:

Noah Smith
Paul Krugman
Bradford Delong
Mark Thoma

Plus the Worthwhile Canadian Initiative folks occasionally drift into progressive territory and are always worth reading. And this is just off the top of my head and including blogs I read regularly. Again: the provocative policy thinker with good ideas and a solid grasp of economists label definitely applies to Yglesias. But I find him a very odd choice for #1 given the alternatives. If anything, I find him awfully centrist on economic matters, at times (which, I suppose, could explain the appeal).

Evaluating evidence

I want to steal a quote from Paul Krugman to illustrate a point:

And in the end, Ryan’s answer is that we need strong economic growth, the kind that we get by cutting taxes on the rich. Because that’s why the Clinton years were an economic disaster and the Bush years so prosperous.

Is this strong evidence?

First of all, we need to consider a number of causal hypotheses:

1) Tax rates on the rich are unrelated to economic growth
2) Higher tax rates on the rich increase economic growth
3) Economic growth makes it easier to tax the rich
4) Higher tax rates on the rich decrease economic growth

Then we need to consider lags between tax policy and changes in economic growth. I am suspicious of anyone who says that this is an easy problem. After all, what we really want (the counterfactual of what would happen if Bush/Clinton not changed tax policy) is completely unavailable.

So what value is this evidence?

It does rule out one very clear talking point in the debate. It suggests that moderate changes in tax policy (Bush Tax cuts) do not have a stronger effect on economic growth than the economic fundamentals do. We may even take this as weak evidence of hypothesis #4 above (with all of the caveats about not being able to make a strong inference).

So the ideas that tax cuts [focused on high income earners] are a good response to short term problems with weak economic growth seems to be contrary to the best evidence available. Nor does looking at period like the 1950' (with very high marginal rates and rapid growth) seem to provide a lot of support for Hypothesis #2.

But if it is case that Hypothesis #2 is true, we know that it is unlikely to overcome other economic issues (or it would have made the Bush years a time of prosperity). Or, in other words, that the overall effect size of this tax policy change is small relative to other factors (if it works in the direction predicted by Hypothesis #2). Now one can reframe this as a moral question, and some do.

But it is worth considering that, in the absence of controlled experiments, how do we update our expectations when a strategy that sounds reasonable doesn't seem to give expected results.

Why aren't you reading the incidental economist?

Because if you care about health care, they are one of the most informative blogs around for those of us in the medical research community.

Consider this statistic:

By 2010, more than 60% of people lived in areas where insurance premiums cost at least 20% of their income. And that’s just premiums; it doesn’t include deductibles, it doesn’t include co-pays, and it doesn’t include co-insurance.

This is likely unsustainable. The growth rate of insurance is far above that of wages, meaning that health care costs are going to consume a higher and higher percent of people’s incomes in the future. Moreover, this is a problem of the non-elderly. Because of Medicare, few elderly have premiums which consume this level of income.

This statistic very nicely frames the entire underlying issue with the explosion in medical costs. Placed in such stark terms, the question shifts from "can we reduce medical costs" to "how are we going to reduce medical costs".

Hard work

Noah Smith has a couple of interesting posts up, but the one that I really found interesting was "Why conservatives can't get people to work hard". It had several insightful comments including the classic:

One basic idea is that hard work should be rewarded. Obvious, right? I mean, we're supposed to be economists here! People respond to incentives, and they are risk averse. A winner-take-all society is not very conducive to hard work; I'm not going to bust my butt for 30 years for a 1% shot at getting into The 1%. But I am going to bust my butt for 30 years if I think this gives me a 90% chance of having a decent house, a family, some security, a reasonably pleasant job, a dog, and a couple of cars in my garage. An ideal middle-class society is one in which everyone, not just anyone, can get ahead via hard work.

Even more interesting, he points out the underlying ambivalence among conservatives as to whether hard work has a causal link to productivity:

Conservatives, meanwhile, are all too often divided on whether they actually believe that hard work works. Plenty of conservatives have undermined Cowen's hard-work-and-discipline bloc by saying that success in life is all due to natural differences in ability. These "I.Q. conservatives" see inequality as the natural order of things. They have focused on getting people to accept their place in society and learn to live with what they have, rather than strive to move up in the world. This is a very Old British sort of conservatism, a nobility-and-peasants ethos dressed up in the faux modernism of psychometric testing.

Conservatives need to look in the mirror and ask themselves: "Do we really want people to work hard and be disciplined? Or do we just say that in order to keep the peasants from getting restless, when deep down we believe that it's all about good genes?" Because if it's the former, conservatives should do some hard thinking about what actually gets people to work hard. And they should think about how to respond to those among their colleagues for whom it is simply the latter.

I think that the "I.Q. conservatives" (as Noah calls them) are actually a fairly concerning movement. We all know that social structures based on accepting one's lot in life (think feudalism) have shockingly low levels of productivity. A social creed that suggests that this lack of productivity is due to innate personal differences is also one that cannot address any social dysfunction that may be present. After all, if the reason person A is successful is that they are the "right sort of person" then we don't have to handle questions like "why is person B unsuccessful".

A broad adoption of this ethos would be an unfortunate outcome for any society because it then concentrates decision making ability into a more and more restricted class. Democracy and capitalism succeed by making the information base broad. It's not that they always succeed in creating good outcomes. But the track record of a narrow elite making decisions is . . . poor.

Laboratory animals in biomedicine -- a recycled reply

In response to Joseph's recent post on the over-reliance on mice in medical research (which was prompted by this thought-provoking piece in Slate), I thought I'd dredge up something I wrote on the subject last year:

Landscapes and Lab Rats

In this post I discussed gradient searches and the two great curses of the gradient searcher, small local optima and long, circuitous paths. I also mentioned that by making small changes to the landscape being searched (in other words, perturbing it) we could sometimes (with luck) improve our search metrics without significantly changing the size and location of our optima.

The idea that you can use a search on one landscape to find the optima of a similar landscape is the assumption behind more than just perturbing. It is also the basis of all animal testing of treatments for humans. This brings genotype into the landscape discussion, but not in the way it's normally used.

In evolutionary terms, we look at an animal's genotype as a set of coordinates for a vast genetic landscape where 'height' (the fitness function) represents that animal's fitness. Every species is found on that landscape, each clustering around its own local maximum.

Genotype figures in our research landscape, but instead of being the landscape itself, it becomes part of the fitness function. Here's an overly simplified example that might clear things up:

Consider a combination of two drugs. If we use the dosage of each drug as an axis, this gives us something that looks a lot like our first example with drug A being north/south, drug B being east/west and the effect we're measuring being height. In other words, our fitness function has a domain of all points on our AB plane and a range corresponding to the effectiveness of that dosage. Since we expect genetics to affect the subjects react to the drugs, genotype has to be part of that fitness function. If we ran the test on lab rats we would expect a different result than if we tested it on humans but we would hope that the landscapes would be similar (or else there would be no point in using lab rats).

Scientists who use animal testing are acutely aware of the problems of going from one landscape to another. For each system studied, they have spent a great deal of time and effort looking for the test species that functions most like humans. The idea is that if you could find an animal with, say, a liver that functions almost exactly like a human liver, you could do most of your controlled studies of liver disease on that animal and only use humans for the final stages.

As sound and appealing as that idea is, there is another way of looking at this.

On a sufficiently high level with some important caveats, all research can be looked at as a set of gradient searches over a vast multidimensional landscape. With each study, researchers pick a point on the landscape, gather data in the region then use their findings to pick their findings and those of other researchers to pick their next point.

In this context, important similarities between landscapes fall into two distinct categories: those involving the positions and magnitudes of the optima; and those involving the search properties of the landscape. Every point on the landscape corresponds to four search values: a max; the number of steps it will take to reach that max; a min; and the number of steps it will take to reach that min. Since we usually want to go in one direction (let's say maximizing), we can generally reduce that to two values for each point, optima of interest and time to converge.

All of this leads us to an interesting and somewhat counterintuitive conclusion. When searching on one landscape to find the corresponding optimum of another, we are vitally interested in seeing a high degree of correlation between the size and location of the optima but given that similarity between optima, similarity in search statistics is at best unimportant and at worst a serious problem.

The whole point of repeated perturbing then searching of a landscape is to produce a wide range of search statistics. Since we're only keeping the best one, the more variability the better. (Best here would generally be the one where the global optimum is associated with the largest region though time to converge can also be important.)

In animal testing, changing your population of test subjects perturbs the research landscape. So what? How does thinking of research using different test animals change the way that we might approach research? I'll suggest a few possibilities in my next post on the subject.

The limitations of laboratory animals in biomedicine

There is a very interesting take in Slate on the reliance of Mouse Models in biomedical science. You can see the temptation in a fast growing animal that elicits limited sympathy from the general public. But the concern with missing key drugs is real -- especially when the diseases in mice operate differently than in humans.

But the real kicker was the discovery that control mice are overfed, under-stimulated and obese. This puts an entirely new spin on studies that restrict caloric intake. They may be saying that eating a normal diet is life enhancing and not caloric restriction. These sorts of blind alleys can have enormous consequences.

Go and read -- it is worth it.