More bad statistics from Freakonomics

[Caveat: It is difficult to distinguish between bad and badly presented statistics. The reader should always allow for the possibility of a sound process terribly explained. It is possible that Dr. Sacerdote made some attempt to control for the factors mentioned below, but, just to put things in context, researchers in numerous fields (including epidemiology) are currently struggling with this problem with limited success. If Sacerdote has actually cracked this nut, Freakonomics is certainly burying the lede.]

Freakonomics Radio is not helping Marketplace's average.

Dubner: Let me let you hear from a different economist, his name is Bruce Sacerdote, he's at Dartmouth. He wanted to know how much this kind of activist-parenting -- if you want to call it that -- actually pays off. And one way to measure this, especially if you're talking about educational achievement -- which is what parents probably care about the most -- is to look at adoption studies, where you can actually measure the impact that a family, that the parents, will have on a kid.

Ryssdal: So what's his thesis, that kids adopted into, I guess, high-education homes will be more likely to go to college, is that the deal?

Dubner: Exactly right. If parents are so important, then parents can take an adopted kid who might otherwise not have gone to college, and that kid will become college material. So Sacerdote sliced and diced a lot of good data and he did find parental influence.

Bruce Sacerdote: But it's not quite as big as I expected to find.

Ryssdal: All right, so quantify for me: how big is not so big?

Dubner: If you're a child who's adopted into a high-education family -- that is where the parents both went to college -- you are about 16 percentage points more likely to go to college than a kid who gets adopted into a low-education family. So that sounds pretty good, OK?

Until you compare that to the rate for biological kids from high-education families, who are about 75 percentage points more likely to go to college than biological kids from low-education families. So on the one hand, this is a little dispiriting for parents. We don't seem to have as much influence as we might think. On the other hand, in a weird way, it kind of takes some of the pressure off, right? At least it did for Bruce Sacerdote.

Sacerdote: This notion that genes are really important and that kids are hard-wired to do certain things, I think understanding that did help me relax and not worry so much that I was going "screw them up" in some terrible way.

One of the concerns I have about sending an economist to do a statistician's job is that economists, by training, have a tendency to make things simpler.* This works reasonably well in economics where the problems are generally too complex to tackle without making some simplifying assumptions about linearity and additive effects and independence and where (perhaps more importantly) the economists have a pretty good idea of which corners can be safely cut. Unfortunately, when they move outside of familiar ground, you often get something like this.

Unless something important has been left out (always a possibility when something is presented to a general audience), Sacerdote has made the common but serious mistake of treating a select group as a random sample. People who choose to adopt are not a random sample of parents, particularly given the difficulties in the process, and those who do self-select have to go a difficult approval process.

Think for a moment about how unrepresentative the resulting sample is. Now consider traits that are likely to be correlated to both parents' and children's education levels and which the process is likely to select for or select out. Here are a few that come immediately to mind:

Teen parenthood;

Unplanned parenthood;

Poverty;

Drug abuse;

Career level (high income may not be a requirement but financial stability is and these days that represents a significant career accomplishment);

Encouragement and support**;

Educational expectations for their children (I can't imagine that all that many prospective parents who don't give a damn about their kid's education make it to the end of the process).

The genetic component of this question is confounded with all these things and more. As a result, (at least as presented here) Sacerdote actually succeeds in proving the opposite of his conclusion. His example doesn't show what, if any, role genetics plays in how likely a kid is to go to college. That 75 could be mainly the result of poverty (which is highly correlated to parent's education levels). I'm not saying that it is but there is no way of ruling out that and other explanations.

What Sacerdote has shown is that even when you control not only for genetics but also for most obvious environmental factors, you still got a 16 point bump just from having adoptive parents who both went to college. Under any circumstances this would be a substantial improvement but in the context of this natural experiment it's huge.

At the risk of over-sharpening the blade, we have here a natural experiment that shows nothing about the role of genetics in academic achievement. What it does show is that even when you greatly reduce most of the obvious parenting-related environmental sources of variability, you still get a substantial effect from the remaining factors. A highly respected researcher then uses this example to argue publicly that genetic factors far outweigh the impact of parenting.

We really need to do something about Freakonomics.

(I'm also not comfortable with the way Sacerdote gives a percentage difference rather than the ranges but that's a subject for another post)


* There are, of course exceptions. Most notably Nate Silver.

** I'm pretty sure in most low education adoptive homes you won't see this relationship:

Hart and Risley also found that, in the first four years after birth, the average child from a professional family receives 560,000 more instances of encouraging feedback than discouraging feedback; a working- class child receives merely 100,000 more encouragements than discouragements; a welfare child receives 125,000 more discouragements than encouragements.

Is Megan McArdle abandoning the Chicago School?

In the previous post, Joseph directed us to this piece by Megan McArdle. As with almost everything McArdle writes, there was much that was worthy of comment, but this passage struck me as especially interesting:

People really underweight the role that norms play in sustaining a modern economy. I suspect that if the "default option" folks got their way and people started regarding default as a commercial decision with no more moral weight than changing cable vendors, the advocates of this position would be unpleasantly surprised to find that the only thing less fun than being young and burdened with student loans is being young and completely unable to access any form of credit at all.

My first corporate job was building models to decide who got unsecured loans and, based on that experience, I'm pretty sure she's wrong in the particulars. A change in attitudes would affect default rates somewhat but there are a number of powerful external incentives here that would limit the impact of a new set of norms.

I agree, however, with McArdle's larger point, which is strange, since I don't believe that McArdle agrees with it herself.

As noted before, freshwater economics is essentially a reductionist approach that relies heavily on a number of simplifying assumptions. If you claim that social norming plays a significant role in the economy, then you have to allow for local optima and plateaus and collective action problems and all sorts of behavior that is not rational in the economic sense and which will not average out when you look at things in the aggregate. That way lies madness and perhaps even Keynesianism.

It's possible that McArdle is starting to question what she learned at the feet of her Booth School professors, but I think it's more likely that she's willing to engage in a bit of intellectual inconsistency to reach her desired conclusion.

Some reflections on student loan debt

I have a mixed set of feelings about Megan McArdle's Atlantic column. Sometimes I could not disagree with her viewpoints more (mostly when she rails against government spending without a good discussion of what a "public good" really is). Other times I think she does a very good job of thinking through the issues. A recent case in point is her posts on student debt. In these posts she does a really commendable job of pointing out two things that are important:

1) It is very hard to settle student loan debt and the lenders have ruthless policies to singificnatly increase what you owe

2) There is a problem with a social norm that makes default a trivial event

That being said, I must admit that the elephant in the room is probably the increasingly draconian state of US debt law. The stories of police raids over student loans (actually not owed by the person being put in handcuffs) may be missing important facts but the general picture is not pleasant.

But, at a more prosaic level, the inability to fail has a lot of serious issues. I have little sympathy for Elie Mystal , per se, but I do see a broad culture of making things worse when one experiences adverse life events. If one should end up unemployed they may well have no health insurance, limited access to bankruptcy for large debts (student loans), and have issues with basic necessities.

I am all for civic virtue and personal responsibility. These are both things that I wish we had more of. But I worry that we are creating a culture of cascading failures where one thing going wrong can set off a series of disasters. In a world with long term unemployment, is this really a good direction to be going?

What fun is it firing people if you don't get to humiliate them too?

I am at a loss to find any other explanation for this.

Rhode Island does, however, offer a bright side for educators and job security. If you can make it past principal, you seem to be safe. Take Frances Gallo and Victor Capellán, two administrators overseeing a school district so small it only includes one high school, one middle school and a few elementary schools. Despite this high administrator to school ratio, even the most basic aspects of administration (such as scheduling substitutes) are routinely screwed up.

Gallo and Capellán admit the schools that they are responsible for are terrible, but they don't hold themselves responsible. If you can't guess who they do blame, you haven't following the education reform movement.

Some products simply demand a targeted approach

From Yahoo:

Even the most die-hard users admit to having bought some turkeys. Kris Kendall, a marketing manager in Freemont, Calif., has been labeled a "Groupon pimp" by her buddies, since she buys several offers every week and is constantly emailing her friends about terrific deals. But the occasional missteps have cost her serious money.

... "I've also let a few expire because I just never had the mojo to go -- namely, a pole-dancing workout class which I bought for a bunch of my friends, and then we never actually went."

Weekend Gaming -- in memoriam/elegant games

[It does not bode well when you need to make time Monday to catch up on things you were supposed to do over the weekend.]

Toy Designer Ned Strongin Dies at 91

Strongin was known for designing and licensing several award-winning toys, including Connect Four, Weebles, Giggle Wiggle and the Wiggly Giggly ball. Connect Four, his most commercially successful invention, remains one of Hasbro's best-selling games.

"Ned was regarded in the toy and game industry as one of the three post-World War II 'fathers' of external toy and game inventions, along with Marvin Glass and Eddy Goldfarb," said Phil Orbanes, president of Winning Moves Games. Orbanes worked with Strongin to license Giggle Wiggle to Hasbro Games.

If you're not familiar with the Connect Four, here's the introduction from Wikipedia:

Connect Four (also known as Four Up, Plot Four, Find Four, Four in a Row, and Four in a Line) is a two-player game in which the players first choose a color and then take turns dropping their colored discs from the top into a seven-column, six-row vertically-suspended grid. The pieces fall straight down, occupying the next available space within the column. The object of the game is to connect four of one's own discs of the same color next to each other vertically, horizontally, or diagonally before one's opponent can do so.

It's an entertaining and genuinely challenging game and it illustrates the most fascinating attributes of what I've called elegant games: a simple rule or condition (in this case, that each column must be filled from the bottom up) can greatly increase the complexity and subtlety of a game.

A Groupon postscript

Felix Salmon directs us to Kevin Kelleher's memorable profile of Groupon's co-founder and chairman.

FORTUNE -- "Lets start having fun... lets get funky... let's announce everything... let's be WILDLY positive in our forecasts... lets take this thing to the extreme... if we get wacked [sic] on the ride down-who gives a shit... THE TIME TO GET RADICAL IS NOW... WE HAVE NOTHING TO LOSE..."

This is a quote from the dot-com era. It's pretty much what you'd expect a novice executive to say back then, when it was all about money and not at all about creating something good. It was written in an email by the co-founder of a company called Starbelly.com, which labeled itself a B2B provider -- back when people greeted that phrase with a straight face.

In early 2000, Starbelly sold itself to another company called Ha-Lo Industries for $240 million, much of which went to the author of those words, a man named Eric Lefkofsky. Not long after that transaction, Ha-Lo declared bankruptcy. Shareholders and others blamed the Starbelly deal, and a series of lawsuits ensued.

After that, it gets even more interesting.

More on McKinsey

As Joseph already mentioned, the recent report from McKinsey from the economic impact of health care reform has generated a lot of controversy. Now, courtesy of TPM, we learn that some of that controversy extended to the organization itself.

The predictable fallout led Democrats, and several reporters, to press McKinsey for the survey itself -- a request McKinsey has declined on the grounds that the material is proprietary.

But multiple sources both within and outside the firm tell TPM the survey was not conducted using McKinsey's typical, meticulous methodology. Indeed, the article the firm published was not intended to give the subject matter the same authoritative treatment as more thorough studies on the same topic -- particularly those conducted by numerous think tanks, and the Congressional Budget Office, which came to the opposite conclusion. And that's created a clamor within the firm at high levels to set the record straight.

"This particular survey wasn't designed in away that would allow it to be peer review published or cited academically," said one source familiar with the controversy.

All sources were granted anonymity, in order to be able to speak candidly about the controversy.

Reached for comment today, a McKinsey spokesperson once again declined to release the survey materials, or to comment beyond saying that, for the moment, McKinsey will let the study speak for itself. However, McKinsey notes that the survey is only one indicator of employers' potential future actions -- that the conclusions remain uncertain and employers' future decisions will ultimately depend on numerous variables. The three authors of the report were not immediately available for comment.

Another keyed-in source says McKinsey is unlikely to release the survey materials because "it would be damaging to them."

Both sources disagree with the results of the survey, which was devised by consultants without particular expertise in this area, not by the firm's health experts.

A third source speculates that the firm may have reached its outlying conclusion by basing its questions on the firm's own advice to clients on how best to arbitrage the new reforms. Specifically, under the law, employers could devise their benefits packages in ways that makes them unappealing to lower-income employees, who would then have to enter the exchanges. Though TPM could not confirm this, the conclusion is supported by a disclosure within the McKinsey study itself.

Reporting on methods

Austin Frakt has the definitive take on the new McKinsey report:

As someone who does research, this really bothers me. It should bother you too. Look, anybody can say what they like on a topic. They can put out a glossy report. They can claim they did a “survey” to make it sound scientifically rigorous. They can talk to the media all about it. They can stand behind their good name and reputation, if they have one. But when what they’re saying runs counter to previous experience and other credible estimates, they’d better have a good explanation.

But, McKinsey has no explanation. None. They’re stonewalling. You know what would happen to me if I tried that? Suppose I sent my new results to a journal, results that were very different from that of others, and said, “Trust me. They’re good.” Well, my paper would be laughed out of the editorial office.

And that’s as it should be. That would not be research. That would be the opposite of research. That would be indistinguishable from making things up. Well, anybody can make things up. The difference between making things up and actually doing some sound, rigorous work is the difference between fiction and reality.

It is no surprise that I, also, see the methods as being critical. Some extremely misleading associations can occur depending on how the sample was developed and how the survery was conducted. Of special concern:

"Our survey shows significantly more interest in alternatives to ESI [employer sponsored insurance] than other sources do, for several reasons," the report says. "Interest in these alternatives rises with increasing awareness of reform, and our survey educated respondents about its implications for their companies and employers before they were asked about post-2014 strategies. The propensity of employers to make big changes to ESI increases with awareness largely because shifting away will be economically rational not only for many of them but also for their lower-income employees, given the law's incentives.

So the survey educated employers on how it would be economically rational to not give health insurance and then 70% of them still decided to do so!! This is the precise opposite of the headline -- that employers will tr to keep covering employees even if it is not an optimal strategy. Probably because they want to do right by their employees.

Why is this key issue not getting a lot more attention in the interpretation of the report? And why is McKinesy's reputation not being brought into question given that they released such a report?

update: The fallout continues.

"We've made huge advances in what they're called"*

I have a feeling we're going to get awfully tired of the word 'social.'

US. American Express launched an integrated marketing campaign today showcasing the possibilities that exist within its Membership Rewards programme, with a focus on demonstrating how points are a ‘social currency’, that when redeemed help cardmembers connect to the people, experiences and things they love.

* For the source of the quote, click here.

The real problem with Groupon

If you've been reading this blog over the past few days you've probably picked up on the fact that neither if us is a big fan of Groupon as an investment. We've talked about issues with priming and adverse selection, the company's oddly primitive approach to targeting and its curious indifference to customer data. We didn't even get around to the difficulties with running a business based on so many merchant partners.

But as serious as these things might be, they are fixable. A management shake-up, a few smart people in the right places. I've seen business lines with bigger problems turned around and by companies with fewer resources than Groupon.

Problems with business plans are different and when you look at the bare bones core of Groupon's business plan, stripped of all the hype and obfuscation and needless complexity, Groupon is based on bringing together customers and merchants and convincing both to do something they normally hate doing: respectively, paying up front and taking a loss.

A Groupon offer is not a coupon; it's what we would normally think of as a gift card, a really crappy gift card at an exceptionally good price. How crappy? It can be used at only at a specific business (sometimes only at one location). You have to buy it on their schedule. Many if not most have expiration dates. Some even have blackouts on holidays. Finally, it can only be used one-per-visit with any unused balance being sacrificed.

People purchase these terrible cards partly because of the considerable (and non-sustainable) level of advertising and buzz; mainly, though, they buy them because the discount, both nominal and effective, is huge. (Felix Salmon has argued that the effective discount can be much smaller, but based on the offers I've received and reports from other users, it's generally easy to keep a purchase close to the offer amount.)

In other words, Groupon has huge revenue because, even when you take into account breakage, its merchant partners are offering huge discounts and certainly, in some cases, taking a loss.

What do merchants get in exchange for these discounts? The stated reason is to bring in new customers, but if that's your goal, Groupon is a terrible choice. Its customer list is enormous but of poor quality and the company hasn't even bothered to gather the data needed to do the most basic of targeting. There are any number of options using the internet and/or traditional media that will bring in a much higher percentage of the right kind of customers.

But merchants keep coming to Groupon despite its mediocre list and the fat slice it takes out of every deal (from Wikipedia):

As of 2010^[update], it is difficult for local merchants to get Groupon interested in agreeing to a particular deal. According to the Wall Street Journal, seven of every eight possible deals suggested by merchants were dismissed by Groupon.

Just to be clear, merchants spend time and effort putting together offers that will probably be rejected and, if they're not, will probably bring in a lot of customers they don't want. They do this because Groupon has successfully branded itself as the next big thing.

This is not something the company stumbled into. Groupon has aggressively pursued fast growth, generated ubiquitous buzz and has done its damnedest to portray itself as part of the social media movement. The 'social' aspect of Groupon's business has always been trivial to the point of cosmetic but it plays a large, even dominant role in the public image of the company.

In one sense, Groupon's strategy has worked very well. After all, the people who started the company will almost certainly walk away with a great deal of money. Eventually, though, the company will have to make the transition to former next big thing and since being the current next big thing is an essential part of the company's model, that transition is not going to be pretty.

(Probably) OE's penultimate Groupon

From the the easily impressed Wall Street Journal:

Yet for Mr. Lefkofsky, Groupon wasn't a one-off in the lottery of high-tech wealth. It was an extension of a lifetime of starting and selling companies-- sometimes with mixed results. The process has led him to a set of guiding business principles: Enter big, fast-growing markets, change course when things aren't working and use data as your guide.

"In our current environment, business and customers are changing so much faster than in the old days," he said in an interview. "You need access to information to figure out what to do next."

Just for future reference, a lot of fast moving, information-based companies start collecting data on customers as soon as they launch.




(I'm still having trouble believing this is a new feature.)

How exactly do you cold call a CT scan?

Via Chait, M.S. at the Economist has some sharp thoughts on the way markets handle health care.

Here's one example among a million. The other day I went to the IPO announcement of a company that does some fairly state-of-the-art medical stuff. The company was spun off from a public institute a few years back to exploit this technology, but it's been unable to establish significant revenue or market share, or to get within shouting distance of breaking even. Meanwhile, competitors with similar technologies have gobbled up most of the market share, and one is already quite profitable. The company said it planned to raise some tens of millions of dollars with the share issue, many times its current annual expenditures and about a third of its overall market cap. And what would it do with this money? It was going to use half of it to finance a marketing drive, targeting key decisionmakers at American health-care providers and health insurers, and doctors.

Why hadn't this company been able to generate significant revenues? Were its technologies inferior? No, said an independent molecular biologist I talked to. Its product was certainly as good as the competition's. Moreover, it had actually gone to the trouble of getting its technology approved by the FDA, which the competition hadn't. (In this sub-sector FDA approval isn't yet mandatory.) But it hadn't marketed itself well. It hadn't established the relationships with providers and insurers that would ensure that its product was the one they selected. Doing so would require a marketing budget of tens of millions of dollars, in a sub-sector where the entire annual market is a few hundred million dollars.

Just think about this for a minute. A medical technology company is going public to generate the money it needs to advertise its products to hospital directors and insurance-company reimbursement officers. This entails significant extra expenditures for marketing, the new stocks issued to fund the marketing will ultimately have to pay dividends, banks will have to be paid to supervise the IPO that was needed to generate the funds to finance the marketing campaign (presumably charging the industry-cartel standard 7%)...and all this will have to be paid for by driving up the price the company charges to deliver its technologies. But beyond the added expense, why would anyone think that a system in which marketing plays such a large role is likely to be more effective, to lead to better treatment, than the kind of process of expert review that governs grant awards at NIH or publishing decisions at peer-reviewed journals? Why do we think that a system in which ads for Claritin are all over the subways will generate better overall health results than one where a national review board determines whether Claritin delivers treatment outcomes for some populations sufficiently superior to justify its added expense over similar generics? What do we expect from a system in which, as ProPublica reports today, body imaging companies hire telemarketers to sell random people CT scans over the phone?

Canadian Health Care

Aaron Carroll provides support for the Canadian Health Care system after comments by Paul Krugman. It is a pretty devastating critique (quality seems slightly higher) made even more so when you consider cost and universal access.

The universal access point is important. When people talk about fear and uncertainty (for business) as being a concern, I think it could also be an issue at the individual level. After all, the fear of an unexpected dread disease appears to be a major concern for most people. How much better would it be if we could eliminate these concerns by breaking the link between health coverage and employment while controlling costs in an effective manner?

The most startling thing in Dr. Carroll's graphs was that physicians are beginning to emigrate to Canada (on net). So even the medical care specialists (with requisite wage controls) seems to like the system.

Cap'n Crunch -- the Movie (you know they'll get around to it eventually)

In the progression of movies-based-on-old-TV-shows to movies-based-on-theme-park-rides to movies-based-on-board-games (via About.com), this would seem to be the next logical step.

Constantine von Hoffman summarizes the latest crop of films based on consumer goods:

Universal is set to make Ouija with the single-named director McG (Terminator Salvation) in line to join TRON: Legacy writers Edward Kitsis and Adam Horowitz — not the guy from the Beastie Boys, who’s Adam Horovitz, with a “v”. And this is all under the watchful eye of Michael Bay’s

• production company.
  
• Ridley Scott is developing Monopoly. Yes, the Ridley Scott who made Aliens and Gladiator. Lord, I wish I could make up stuff this good.
• Jonathan Aibel and Glenn Berger, who co-wrote that timeless classic, Kung-Fu Panda 2, have said they are working on the Candy Land script. No, a gun was not being held to their heads when they said this. What a strange question.

The back story here (which is much more interesting than any of these movies are likely to be) involves innovation, demographic shifts, winnowing processes, business psychology and a truly nasty collective action problem, but the upshot is a situation where name recognition of any kind is given far more weight by studio executives than can be justified by modelling or common sense.

On the bright side, a Cap'n Cruch movie should produce a nice check for the estate of Jay Ward.