“We've made enormous advances in what they're called” -- more on corporate data cooking

Yesterday, I mentioned how bundled offers and the ability to pick the most advantageous data could allow a company to produce any number of grossly dishonest statistics. Today over at Baseline Scenario, James Kwak explains how J.P. Morgan can use acquisitions and flexible definitions to perform similar magic with its promise to loan $10 billion to small businesses:

Still, $10 billion is still an increase over the previous high of $6.9 billion in 2007, right? Well, not quite. Because in the meantime, JPMorgan Chase went and bought Washington Mutual. At the end of 2007, Washington Mutual held over $47 billion in commercial loans of one sort or another (from a custom FDIC SDI report that you can build here). Most of those are not small business by JPMorgan’s definition, since commercial real estate and multifamily real estate got put into the Commercial Banking business after the acquisition. But that still leaves $7.5 billion in potential small business loans, up from $5.1 billion at the end of 2006, which means WaMu did at least $2.4 billion of new lending in 2007.

I don’t know how much of this is small business lending, but this is part of the problem — banks can choose what they call small business lending, and they can choose to change the definitions from quarter to quarter. It’s not also clear (from the outside, at least) what counts as an origination. If I have a line of credit that expires and I want to roll it over, does that count as an origination? My guess is yes. Should it count as helping small businesses and the economy grow? No.

Sitting and obesity

It's one of the more difficult epidemiology questions to answer: why is obesity rising so quickly?

This is a very hard question to answer decisively, there is some reason that Americans have gotten over-weight in the past 30-40 years. It's not pure food abundance as we have had that for a long time. It's not genetic in the sense of the population genetics changing as there has not been enough time (genetic susceptibility is another matter).

So the idea that more time spent sitting leads to obesity is a very interesting hypothesis. I wonder how feasible it would be to design a cluster randomized trial for workplace interventions (like standing to use the computer).

Avandia to be withdrawn?

From Derek at In the Pipeline, it looks like leaks from a Senate report indicate that Avandia is about to be removed from the market. Thus ends a long run of pharmacoepidmeiology papers on the subject. It's not been an area that I worked in personally, but some of my friends have. Studying the heart risks of Avandia is tricky for observational data -- the disease being treated (diabetes) is a risk factor for the major side effect. This makes it very hard to separate disease and drug effects (especially since it is hard to control for severity and duration of a silent disease like diabetes).

But the existence of a comparator drug that showed a better risk profile for cardiovascular events was probably the decisive factor. Pharmacovigilance really can save lives!

How to Lie with Statistics -- Allstate Edition

For our latest statistical lie of the week, check out the following commercial.




At the risk of putting too fine a point on it, here's a full breakdown.

Customers of the two companies fall into one of four categories:

Geico customers who would get a better deal with All State;

Geico customers who would get a better deal with Geico;

All State customers who would get a better deal with All State;

All State customers who would get a better deal with Geico.

If we knew the relative sizes of those four groups and the average savings of the first and last groups we'd have a fairly comprehensive picture. Not surprisingly neither Allstate nor GEICO went that far. Both companies talk about the savings of people who switched.

Most people presumably switch providers to get a better deal (putting them in the first or last groups). Furthermore, switching is a hassle so the savings have to be big enough to make up for the trouble. The result are highly biased self-selecting samples of the first and last groups.

When GEICO simply mentions a potential savings of 15%, they are being a bit less than forthcoming but the claim that you might be able to save a substantial amount of money by switching is reasonable. For honest-to-goodness lying you need to wait for the Allstate commercial.

Allstate also bases their claims on the savings of those who switched to their company, but unlike GEICO they use those claims as part of a classic lie-by-hypothesis -- making a statement then supporting it with an incomplete or unrelated statistic. The ad starts with a trustworthy-sounding Dennis Haysbert saying "If you think GEICO's the cheap insurance company, then you're going to really be confused when you hear this" then touting an average savings of $518.

Yes, you might be confused, particularly if you don't realize the sample is ridiculously biased or that we aren't told the size of the policies or how long a period the $518 average was calculated over (the small print at the bottom refers to 2007 data which seems a bit suspicious, particularly given the following disclaimer at the bottom of Allstate's website "*$396 Average annual savings based on information reported nationally by new Allstate auto customers for policies written in 2008." No competitor is mentioned so the second number is presumably a general average. This could explain the difference in the numbers but not decision to shift periods).

I would also be suspicious of the data-cooking potential of Allstate's bundled products. Here's how the old but effective scam works: you single out one product a loss leader. They may sell this as a feature -- save big on car insurance when you get all of your coverage from Allstate -- or the numbers may be buried so deeply in the fine print that you have no idea how your monthly check is being divided. Either way this gives the people massaging the data tremendous freedom. They can shift profits to areas that Wall Street is excited about (happens more often than you might think) or they can create the illusion of bargains if they want to counter the impression of being overpriced. I don't know if any of this is going on here but I'm always cautious around numbers that are this easy to cook.

I would also take into account Allstate's less than shining reputation in the insurance industry, particularly regarding the company's strategies since the mid-Ninties. The story has been covered by Business Week, PBS and Bloomberg which supplied the following:

One McKinsey slide displayed at the Kentucky hearing featured an alligator with the caption ``Sit and Wait.'' The slide says Allstate can discourage claimants by delaying settlements and stalling court proceedings.

By postponing payments, insurance companies can hold money longer and make more on their investments -- and often wear down clients to the point of dropping a challenge. ``An alligator sits and waits,'' Golden told the judge, as they looked at the slide describing a reptile.

McKinsey's advice helped spark a turnaround in Allstate's finances. The company's profit rose 140 percent to $4.99 billion in 2006, up from $2.08 billion in 1996. Allstate lifted its income partly by paying less to its policyholders.
...
Allstate spent 58 percent of its premium income in 2006 for claim payouts and the costs of the process compared with 79 percent in 1996, according to filings with the U.S. Securities and Exchange Commission.

So, even if we put aside the possibility of data cooking, we still have an ethically tarnished company dishonestly presenting a meaningless statistic and that's good enough for our statistical lie of the week.

The Tuition Paradox

This post and Joseph's follow-up has gotten me thinking about a strange aspect of the economics of higher education in recent decades.

At the risk of oversimplifying, undergraduates are primarily paying for instruction and evaluation. The school will teach the student a body of knowledge and a set of skills and will provide the student with a quantitative measure (backed by the reputation of the school) of how well he or she mastered that knowledge and those skills.

The costs associated with providing those services is almost entirely labor driven. While there are exceptions (particularly involving distance learning), most instructors use minimal technology and many just rely on the white board. This is not a criticism (A good teacher with a marker always beats a bad teacher with a Powerpoint), but the costs of a service that can be provided with simple facilities and little or no specialized equipment will always be labor driven.

Twenty or thirty years ago, when you took an undergraduate class you were likely to be taught by a full-time faculty member, not someone with a high salary but reasonably well paid professional with good benefits and excellent job security. These days you are far more likely to be taught by a badly paid adjunct with no benefits or job security.

In other words, when you take into account inflation, the cost to universities of providing instruction and evaluation have dropped sharply while the amount universities charge to provide these services has continued to shoot up.

I'm not say that this is all a scam or that administrators are out there stuffing their pockets, but I do think there's something wrong with this picture.

Are humanities and science careers different?

Mark pointed me to an article by Thomas H. Benton about graduate school in the humanities. These issues have been persistent concerns in the field; I recall arguing about the job prospects of humanity graduates as an undergraduate philosophy major. I think that there really is an argument that the costs (in tuition, living expenses and so forth) that are required for an advanced degree in the humanities can't possibly be compensated for by post-degree job prospects.

Which is okay, if the goal of the degree is edification. But these degrees are not often marketed as expensive luxury goods . . .

In science, I think we are better off. We train people with marketable skills that can lead to careers. Post-degree placement is considered an important metric of success. But I think tales like this are a call to action to make sure that we continue to provide relevant training and to be cautious about blurring the distinction between data and anecdote in terms of outcomes.

If nothing else, it seems to be a good case for outcomes tracking . . .

Academic work hours

It is pretty true that academics is not a Monday to Friday job. However, there is actually a nice compensation that can often happen. When I was at McGill I made some very good friends just by being in the lab at odd hours (especially late at night). There can be a sense of shared struggle that is an overlooked bonus. Of course, it'd have been even nicer if there was a late night office shop to take breaks in but you cannot have everything!

Multiple Testing

Interesting. False positives in popular fields appear to be much more strongly driven by the number of groups testing these hypotheses rather than by fiddling with data. A very comforting result, insofar as it is true.

More troublesome, is that it is unclear what we can do about it. Being better about publishing negative results helps but is never going to be a perfect solution; especially when reviewers may be more skeptical about results that do not match their intuition.

The difficulty of Soft Outcomes

There is currently a movement to ban combination medications with acetaminophen as an ingredient. The reasoning behind this appears to be due to the potential for liver damage caused by excessive doses of the medication. The estimate of 458 deaths per year seems like a lot, until you realize the denominator is not specified (it won't be the entire US population but it might be 10's of millions).

The other issue, and the one that is interesting to an epidemiologist, is the soft nature of the competing risk. The alternatives to acetaminophen is either a narcotic or a non-steroidal anti-inflammatory drug like ibuprofen. Both of these drugs have downsides (addiction, gastrointestinal bleeding) as well.

But the real alternative is less pain control. And that is hard to judge because it is a soft outcome. How much suffering is worth a life? Lives are easy to count but massively reduced quality of life is much, much trickier. But I think it is important to realize that a hard to measure outcome can still have a known and measurable effect on real people.

So I guess what I want to see a clear articulation of what are the alternatives to the current approach to publishing in hot fields.

Post-Doctoral Fellowships

Am I really atypical in having had a decent post-doctoral fellowship? Is it a feature of the University of Washington or of my PI?

But when I read bitter stories about bad experiences then I wonder if this is a "there but for the grace of some powerful entity I go".

I think one issue is that people expect a lot at the end of the PhD (and not without justification -- the PhD is a long and difficult process). But the reality is that the PhD is the license to do research -- meaning you get to start at the beginning all over again. After 12 years of schooling (and an outside career in the financial services industry) that can be rough.

I'm lucky to have found a supportive place and I am pleased that I am moving forward to a good position (although quite sad to be leaving the state of Washington). Here is hoping that academic work is as unexpected pleasant as post-doccing turned out to be!

Cheerios vs the FDA

Via Tyler Cowen, there is this new article.

So, because health research is being cited, Cheerios are a drug? So what is the rationale for regulation of this product? Patient safety? I fear that denying patients food because clinical research has not been done would seem to be an unwise idea.

I do think that poor regulation of products can lead to harm but there does seem to be a difference of kind here. Nor do I think that merely citing research should be an issue: there should be more careful research into diet and not less.

What should your final model be?

In the decision to present the best possible model based on the weight of evidence, should you present the results of Bayesian Model Averaging or of a single selected model based on variables that had strong posterior probabilities in the BMA analysis?

I think you have to present the posterior probabilities. They actually contain MORE information than the selected model. More importantly, the p-values in a selected model don't seem to give a good sense of model uncertainty. This feature is something that I dislike about all automated model selection approaches in that they don't represent competing models that were close to the final model in likelihood but just happened not to be selected.

But it looks like almost no substantive papers use BMA. Why is this?

Measuring changes

Is there anything trickier than measuring changes? Probably, yes, but this has got to be competitive for the difficulty in translating clear concepts into analysis. We can all describe, using ordinary language, the idea that a person may become more physically active with time.

But how do we operationalize this using the tools available in the standard cohort study. Do we look at change? What about change conditional on baseline? But the second quantity is going to have issues with measurement error.

What about categorizing things? Sadly, this creates the annoying problem of regression to the mean and ignores the bigger question of how do we define a meaningful threshold. Let alone the issue of reducing the information content of a continuous variable by breaking it into two categories. This reduction might be defensible if these categories align with clear clinical guidelines and so make results easier to operationalize. But many variables don't have this lucky property.

The more I look at this, the trickier the whole thing becomes.

Mission Creep

Professor in Training has a post on the amazing amount of time that it takes to certify a lab and/or a researcher. I find it very interesting how there has been a slow increase in the amount of overhead to do research. Each step may be both reasonable and justified. But the total effect can be quite counter-productive.

And, I must admit, that I am not altogether clear that it is a better world when the key skill for a junior professor is negotiating bureaucracy and/or finding ways around regulation. I know why making these rules apply to everyone is useful (as people avoid these things if they have any chance at all -- or at least I do). But maybe we should learn to balance total impact on research with protection of lab-workers and research subjects?

Sick Days part 2

A nice follow-up to my post yesterday about the public health issues of poor sick leave policy. This area is a place where there is a bad tendency to spout what good practice is and then put incentives in place that make following such practices highly problematic at best.