Worst Statistic of the Day

Via Lawyers, Guns, and Money comes the piece most detached from reality I've seen this year:

When Josh McFarland graduated from Stanford he owed $40,000 in student loans and couldn't fathom a way he'd ever pay it off and have a future for himself - not unusual for the typical young adult these days. Then he went to work for Google.

As a product manager, he got stock options and cashed them in over the five years he worked there. He married a fellow Google employee, so she had stock too. Then she moved on to Yelp, and he quit to launch TellApart, which provides technology solutions for e-commerce sites. Now 33, McFarland has a 3-year-old and a newborn and no longer has to think about his student loan: His company has $17.75 million in venture capital investment. While he doesn't consider himself retire-now rich, his piece of the company affords him what he calls "breathing room" and what other people might call wealth.

McFarland is on the starting end of Generation Y, the cohort born in the United States after 1980 that is typically portrayed as saddled with massive student debt, underemployed and underpaid. More than a third of the 80 million group of so-called millennials live with their parents, according to the Pew Research Group.

But McFarland is part of the sizeable minority that is doing quite well: 12 million Gen Y-ers make more than $100,000, according to the Ipsos MediaCT's Mendelsohn Affluent Survey. Many of them, in technology fields, live frugal work-based lifestyles and are not saddled with the six-digit student debt held by doctors and lawyers.

Paul Campos researched this statistic and discovered where it came from:

The source of the wrongheaded statistic appears to be this: the cited survey claims 59 million Americans live in households with incomes of $100,000 plus, and that 20% of “affluent consumers” are in the Gen Y cohort (this appears to mean that 20% of the adults who live in households with $100,000+ incomes are of this age). The story’s author then did some very bad math to generate the claim that “12 million Gen Y-ers make more than $100,000.” 

But that is totally different than these people making > $100,000 per annum.  That could also be people who are utterly unemployable living in their parent's basement. Or a trophy wife/husband of a hedge-fund billionaire. Or it could be 5 young adults sharing a house together, all making $22,000 a year as Starbucks' employees. 

At the very least it uses a highly no-representative sample and portrays the wealth of young adults in a completely misleading way.  This does not add to the conversation and it appears in the New York Times.  If only they had a prominent economist on staff who could consult over this type of data . . .

Is smoking really the most important predictor of obesity?

Does this headline really represent the data?:

The No. 1 Reason Americans Are Getting Fatter: We're Not Smoking

The text goes on to discuss the rise of anti-smoking laws and the increase in obesity. Then at the end of the article, they discuss the actual magnitude of the effect:

"Using the traditional Blinder-Oaxaca decomposition technique" -- a social science method often used to study two groups or components with different credentials; a method, essentially, of comparing apples to oranges -- "we find that cigarette smoking has the largest effect: the decline in cigarette smoking explains about two percent of the increase in the weight measures," Baum and Chou, both professors of economics at Middle Tennessee State University and Lehigh University, respectively, explain in the paper's abstract. "The other significant factors explain less."

Two percent? Really? So non-smoking factors explain 98% of the change?

So either the model explains very little of the variation or smoking is a small contributor to the obesity epidemic, regardless of how it ranked on a list of covariates. And this doesn't even consider the possibility of correlation (obesity is rising as smoking falls) of temporal trends instead of causation.

In either case, that is not a useful headline.

Appropriate Scales

Paul Krugman has a clever post about where to start a Y-axis of a graph from. He plots the temperature changes of New York . . . in kelvins. The result is a graph that makes it seem like New York has very small variations across the year.

On the other hand, very small variations can look large if the axes covers only the range of variation. The real piece of information that is missing is what is the size of a meaningful change. Maybe we should have graphs declare this extra piece of information so that this point can be discussed along with the variation shown in the graph, itself,

Math issues

Felix Salmon's last two posts are brilliant. I strongly recommend reading both of these posts. Seriously!

Heck, I am likely to come back and post on them later today if I have enough time. Paul Krugman and Mark Thoma might be representing the outrage of progressives, but Felix is documenting the damage coldly and dispassionately.

However, this is a statistics blog. So I wanted to visit a comment on one of Felix Salmon's posts:

The top 10% of Americans pays 45% of all taxes, a higher proportion by far than any other country, while the American top 10% earns only 33% of total income.

By comparison, the 10% of the UK earns 32% of total income and pays only 38% of all taxes. And the UK is one of the more progressively taxed countries. The taxes paid by the top 10% of other European countries are on average around 30% of total taxes.

As an epidemiologist (a field that is in love with proportions), I realize how misleading they can be. In the example above, the rich earn roughly the same amount of income. But they pay a higher proportion of taxes. However, is it the proportion of taxes that matters or the absolute tax burden?

Consider a simple example. In two countries there are 10 people. Of these people, nine make 10 units of income per year and one makes 50 unites of income. This is pretty close to the income distribution of the US and UK cited above.

In country A, the nine people pay 2 units of tax per year and the rich person pays 16 units of tax. Total tax revenues are 34 units (or 24% of GDP). The rich definitely pay more as a proportion of taxes but pay 47% of the taxes overall.

In country B, they pay something more like the OECD average (35%). So the nine people pay 3.5 units of tax (31.5) and the rich pay 17.5 units of tax. That is 49 units of tax collected (exactly 35%). The rich pay 36% of the tax burden.

Which country is a low tax country for rich people to live in? One where they pay an effective tax rate of 32% versus 35%? Or one where they pay a higher proportion of total taxes collected?

The trick here is that the US total tax burden is so light that the top 10% can both pay a higher proportion of taxes than in other countries and, at the same time, still pay less absolute tax. That can make proportional analysis potentially misleading.

How to REALLY lie with statistics

Darrell Huff had a witty explanation of how you can change the impression a graph gives by playing around with the scale and range of the y-axis, but even in a book called How to Lie with Statistics he never even considered changing just part of the scale of an axis. Of course, Huff was limited by the fact that his book was based on real statistical lies and screw-ups that he had found in the popular press as of the early Fifties. He also tried to limit himself to lies that were at least true in some technical sense. There was no way he could have anticipated Fox News.

The catch comes from Media Matters via Mark Thoma.


Here is the same graph with vertical lines for full comic effect.


Go ahead, measure them for yourself. It's fun.

In all fairness to Fox, the period from September '08 to March '09 did feel like a long time.

How to Lie with Statistics -- Allstate Edition

For our latest statistical lie of the week, check out the following commercial.




At the risk of putting too fine a point on it, here's a full breakdown.

Customers of the two companies fall into one of four categories:

Geico customers who would get a better deal with All State;

Geico customers who would get a better deal with Geico;

All State customers who would get a better deal with All State;

All State customers who would get a better deal with Geico.

If we knew the relative sizes of those four groups and the average savings of the first and last groups we'd have a fairly comprehensive picture. Not surprisingly neither Allstate nor GEICO went that far. Both companies talk about the savings of people who switched.

Most people presumably switch providers to get a better deal (putting them in the first or last groups). Furthermore, switching is a hassle so the savings have to be big enough to make up for the trouble. The result are highly biased self-selecting samples of the first and last groups.

When GEICO simply mentions a potential savings of 15%, they are being a bit less than forthcoming but the claim that you might be able to save a substantial amount of money by switching is reasonable. For honest-to-goodness lying you need to wait for the Allstate commercial.

Allstate also bases their claims on the savings of those who switched to their company, but unlike GEICO they use those claims as part of a classic lie-by-hypothesis -- making a statement then supporting it with an incomplete or unrelated statistic. The ad starts with a trustworthy-sounding Dennis Haysbert saying "If you think GEICO's the cheap insurance company, then you're going to really be confused when you hear this" then touting an average savings of $518.

Yes, you might be confused, particularly if you don't realize the sample is ridiculously biased or that we aren't told the size of the policies or how long a period the $518 average was calculated over (the small print at the bottom refers to 2007 data which seems a bit suspicious, particularly given the following disclaimer at the bottom of Allstate's website "*$396 Average annual savings based on information reported nationally by new Allstate auto customers for policies written in 2008." No competitor is mentioned so the second number is presumably a general average. This could explain the difference in the numbers but not decision to shift periods).

I would also be suspicious of the data-cooking potential of Allstate's bundled products. Here's how the old but effective scam works: you single out one product a loss leader. They may sell this as a feature -- save big on car insurance when you get all of your coverage from Allstate -- or the numbers may be buried so deeply in the fine print that you have no idea how your monthly check is being divided. Either way this gives the people massaging the data tremendous freedom. They can shift profits to areas that Wall Street is excited about (happens more often than you might think) or they can create the illusion of bargains if they want to counter the impression of being overpriced. I don't know if any of this is going on here but I'm always cautious around numbers that are this easy to cook.

I would also take into account Allstate's less than shining reputation in the insurance industry, particularly regarding the company's strategies since the mid-Ninties. The story has been covered by Business Week, PBS and Bloomberg which supplied the following:

One McKinsey slide displayed at the Kentucky hearing featured an alligator with the caption ``Sit and Wait.'' The slide says Allstate can discourage claimants by delaying settlements and stalling court proceedings.

By postponing payments, insurance companies can hold money longer and make more on their investments -- and often wear down clients to the point of dropping a challenge. ``An alligator sits and waits,'' Golden told the judge, as they looked at the slide describing a reptile.

McKinsey's advice helped spark a turnaround in Allstate's finances. The company's profit rose 140 percent to $4.99 billion in 2006, up from $2.08 billion in 1996. Allstate lifted its income partly by paying less to its policyholders.
...
Allstate spent 58 percent of its premium income in 2006 for claim payouts and the costs of the process compared with 79 percent in 1996, according to filings with the U.S. Securities and Exchange Commission.

So, even if we put aside the possibility of data cooking, we still have an ethically tarnished company dishonestly presenting a meaningless statistic and that's good enough for our statistical lie of the week.