Complex Constructs

Andrew Gelman has a very interesting discussion about whether happiness is a U-shaped curve in relation to age. What I found most interesting is how people focused on the "interesting conclusion" (the U-shape) even when there was a broad selection of curves to consider:

At the very least, the pattern does not seem to be as clear as implied from some media reports. (Even a glance at the paper by Stone, Schwartz, Broderick, and Deaton, which is the source of the top graph above, reveals a bunch of graphs, only some of which are U-shaped.)

But what I found the most interesting is that the sub-graphs are on different elements of "well being" (stress, worry,enjoyment happiness, sadness, anger). I wonder if the higher well being among older adults is, in some sense, very different than that of younger adults. Less stress and worry may contribute to increased (overall well being) but it might be a very different positive state than one created by the limitless potential of youth.

So I suppose I wonder if representing a complex vector (as well being has many factors that contribute to it) as a scalar (singl question) might not be eliminating the most useful sources of variability? Even if this approach is the standard in the field, it does not mean that we can't benefit from seeking a more complicated understanding of the phenomenon. I think that Andrew Gelman is on the right track in trying to really understand this complicated (and interesting) relation.

Perils of studying Small Effects

From Andrew Gelman:

For example, suppose you're a sociologist interested in studying sex ratios. A quick review of the literature will tell you that the differences in %girl births, comparing race of mother, age of mother, birth order, etc, are less than 1%. So if you want to study, say, the correlation between parental beauty and sex ratio, you're gonna be expecting very small effects which you'll need very large sample sizes to find. Statistical significance has nothing to do with it: Unless you have huge samples and good measurements, you can pretty much forget about it, whether your p-value is 0.02 or 0.05 or 0.01.

I think that this is correct and ties into a larger narrative: there are a lot of small effects of great theoretical interest that are really hard to show in actual data due to the limitations of realistic sample sizes. Observational epidemiologists often try to handle this by looking at prescription claims databases (for example). But these studies create a new problem: there is a serious concern of unmeasured confounding due to factors that are not measured in these data sources (e.g. smoking, alcohol use, diet, exercise). It's not really a big step forward to replicate lack of power with potential bias due to confounding.

I think the real issue is simply that small effects are difficult to study, no matter how interesting that they are. So I think that Andrew Gelman is right to call for the interpretation of these studies to done in the context of the larger science.

Placebos

This is relevant to Mark's posts on how comparisons between students who win and lose charter school lotteries are similar to open label drug trials. Andrew Gelman quotes Kaiser Fung stating that (in drug trials) the standard comparison is:

Effect on treatment group = Effect of the drug + effect of belief in being treated
Effect on placebo group = Effect of belief in being treated

I think that the mathematical formulation of Mark's concern about using charter school lotteries as follows:

Effect in charter school group = Effect of the charter school intervention + effect of belief in being treated + baseline

Effect on students in standard schools = baseline - (any discouragement effect from losing the lottery)

So if the effect in the charter school students is greater than that in students who lose the lottery and are placed in standard schools, that skill doesn't separate the two sources of variation (placebo versus innate effect of the intervention).

The real test would be to introduce some sort of placebo intervention (a charter school that used standard educational approaches??). But that is not an easy thing to accomplish, for obvious reasons. I suspect that this is why psychologists end up having to use deception as part of their studies, despite the ethical quagmires this produces.

UPDATE: missing link fixed

Sentences to ponder

We always talk about a model being "useful" but the concept is hard to quantify.

-- Andrew Gelman

This really does match my experience. We talk about the idea that "all models are wrong but some models are useful" all of the time in Epidemiology. But it's rather tricky to actually define this quantity of "useful" in a rigorous way.

Undiagnosed diseases

One thought that I have often had about prescription claims databases is that we often can't do anything with missing data. If a patient in such a database has undiagnosed hypertension, for example, it's unclear how to handle it. This is in stark contrast to cohort studies where a missing blood pressure reading is a clear case of missing data and straightforward multiple imputation may do wonders.

So I wonder if Andrew Gelman's idea for count data could be adapted for this purpose?

Or would we be buried by an excessive number of assumptions that might be required to make it work in these settings?

I know there's a Schrödinger's cat joke here somewhere

Andrew Gelman has an excellent post up on the contradictory findings of two teams of researchers on the effect child-gender has on parents' political leanings.

Tyler Cowen reports the following claim from sociologists Dalton Conley and Emily Rauscher:

Using nationally-representative data from the [1994] General Social Survey, we [Conley and Rauscher] find that female offspring induce more conservative political identification. We hypothesize that this results from the change in reproductive fitness strategy that daughters may evince.

This surprised me, because less than a year ago, we reported here on a study by economists Andrew Oswald and Nattavudh Powdthavee with the exact opposite finding:

We [Oswald and Powdthavee] document evidence that having daughters leads people to be more sympathetic to left-wing parties. Giving birth to sons, by contrast, seems to make people more likely to vote for a right-wing party. Our data, which are primarily from Great Britain, are longitudinal. We also report corroborative results for a German panel.

Understanding the results (possibly) in terms of "family values"

This is a fun problem: we have two different studies, both by reputable researchers, with opposite results! I took a look at both papers and can't immediately see a resolution, but I will offer some speculations, followed by some scattered comments.

Andrew speculates that the differences in the findings can be explained by the fact that one study looked at the U.S. while the other looked at Britain with some additional data taken from Germany. Commenter DN elaborated further:

Seems to me that this is more likely a "law and order" effect in the US. Families with daughters are more likely to be concerned about 'protecting' their daughters against violent crime. These impacts are either mitigated in Germany and the UK (because of perceptions of the parties) or swamped by other perceived advantages of left-wing parties.

This brings up one of my least favourite practices in bad statistics reporting: generalizing conclusions about attitudes drawn from a specific culture or social group. One example that stayed with me (reported by the ever credulous NYT under the headline Bicycle Helmets Put You at Risk) was that of Ian Walker, a psychiatrist at the University of Bath. Walker, an opponent of helmet laws, put a sensor on his bike and rode with and without a helmet until he had been passed 2,500 times (see the curse of large numbers). To control for potential gender effects he sometimes donned a long wig (to get the full comic effect, check out Walker's picture below).

Walker found vehicles came on average 3.35 inches closer when he was wearing a helmet (for context, the average passing clearance was over four feet).

Putting aside for the moment questions about the methodology of the study and the sweeping conclusions Walker from it, the New York Times article works under the implicit assumption that despite major differences in traffic laws, road conditions, driver etiquette and education, vehicle type and biking culture, findings from that small stretch of English road are equally applicable to American highways.

Blockbusters, Franchises and Apostrophes

More on the economics of genre fiction

The story so far: last week Andrew Gelman had a post on a book that discussed the dominance of best seller lists and suggested that it was due to their increased quality and respectability. I argued that the quality and respectability had if anything decreased (here), posted some background information (here and here) then discussed how the economics of publishing from the late Nineteenth Century through the Post-War era had influenced genre fiction. The following closes with a look at where we are now and how the current state of the market determines what we're seeing at the bookstore.

As the market shrank in the last part of the Twentieth Century, the pay scale shifted to the feast and (mostly) famine distribution of today. (The century also saw a similar shift for musicians, artists and actors.) Non-paying outlets sprang up. Fan fiction emerged (non-licensed use of characters had, of course, been around for years -- Tiajuana bibles being a classic example -- but fan fiction was written for the author's enjoyment without any real expectation of payment). These changes are generally blamed on the internet but the conventional wisdom is at least a couple of decades off. All of these trends were well established by the Seventies.

With the loss of the short story market and the consolidation of publishing, the economics of writing on spec became brutal. Writing and trying to sell a novel represents a tremendous investment of time and energy with little hope of success. By comparison writing on spec in the Forties meant coming up with twelve to fifteen pages then sending them off to twenty or so potential markets. The best of these markets paid good money; the worst were hungry for anything publishable.

The shift from short story to novel also meant greater risk for the publisher (and, though we don't normally think of it in these terms, for the reader who also invested money and time). A back-pages story that most readers skipped over might hurt the sales and reputation of a magazine slightly but as long as the featured stories were strong, the effect would be negligible. Novels though are free-standing and the novel gets that gets skipped over is the novel that goes unsold.

When Gold Medal signed John. D. MacDonald they knew were getting a skilled, prolific writer with a track record artistically and commercially successful short fiction. The same could be said about the signing of Donald Westlake, Lawrence Block, Joe Gores and many others. Publishing these first time authors was a remarkably low risk proposition.

Unfortunately for publishers today, there are no potential first time authors with those resumes. Publishers now have to roll the dice on inexperienced writers of unknown talent and productivity. In response to that change, they have taken various steps to mitigate the risk.

One response was the rise of the marketable blockbuster. The earliest example I can think of is the book Lace by Shirley Conran. If memory serves, Lace got a great deal of attention in the publishing world for Conran's huge advance, her lack of fiction-writing experience, and the role marketing played in the process. The general feeling was that the tagline ("Which one of you bitches is my mother? ") came first while the book itself was merely an afterthought.

More recently we have Dexter, a marketer's dream ("He's a serial killer who kills serial killers... It's torture porn you can feel good about!"). The author had a few books in his resume but nothing distinguished. The most notable was probably a collaboration with Star Trek actor Michael Dorn. The first book in the series, Darkly Dreaming Dexter was so poorly constructed that all of the principals had to act completely out of character to resolve the plot (tip for new authors: when a character casually overlooks her own attempted vivisection, it's time for a rewrite*).

The problems with the quality of the novel had no apparent effect on sales, nor did it prevent the character from appearing in a successful series of sequels and being picked up by Showtime (The TV show was handled by far more experienced writers who managed to seal up almost all of the plot holes).

The point here is not that Darkly Dreaming Dexter was a bad book or that publishing standards have declined. The point is that the economics have changed. Experienced fiction writers are more rare. Marketable concepts and franchises are more valuable, as is synergy with other media. The markets are smaller. There are fewer players. And much of the audience has a troublesome form of brand loyalty.

Normally of course brand loyalty is a plus, but books are an unusual case. If you convince a Coke drinker to also to drink Sprite you probably won't increase his overall soda consumption; you'll just have cannibalization. But readers who stick exclusively with one writer are severely underconsuming. Convince James Patterson readers to start reading Dean Koontz and you could double overall sales.

When most readers got their fiction either through magazines or by leafing through paperback racks, it was easy to introduce them to new writers. Now the situation is more difficult. One creative solution has been apostrophe series such as Tom Clancy's Op Center. Other people are credited with actually writing the books but the name above the title is there for branding purposes.

Which all leads us back to the original question: Why did thrillers become so dominant?

They tend to be easily marketable.

They are compatible with franchises.

They lend themselves to adaptation as big budget action movies.

Their somewhat impersonal style makes them suitable for ghosting or apostrophe branding.

They are, in short, they are what the market is looking for. As for me, I'm looking for the next reprint from Hard Case, but I might borrow the latest Turow after you're done with it.


* "Is that a spoiler?"
"No, sir. It was spoiled when I got here."

p.s. I was going to tie in with a branding situation Slim Jim snacks faced a few years ago but this post is running a bit long. Maybe I'll get back to it later.

The Decline of the Middle (Creative) Class

I suggested in an earlier post that the rise to dominance of the thriller had not been accompanied by a rise in quality and reputation. In this and the next post, I'll try to put some foundations under this claim.

Popular art is driven by markets and shifts in popular art can always be traced back, at least partly, to economic, social and technological developments as well as changes in popular taste. The emergence of genre fiction followed the rise of the popular magazine (check here for more). Jazz hit its stride as the population started moving to cities. Talking pictures replaced silents when the technology made them possible.

Crime fiction, like science fiction first appeared in response to demand from general interest magazines like the Strand then moved into genre specific magazines like Black Mask and a few years later, cheap paperbacks. The demand for short stories was so great that even a successful author like Fitzgerald saw them as a lucrative alternative to novels. There was money to be made and that money brought in a lot of new writers.

It seems strange to say it now but for much of the Twentieth Century, it was possible to make a middle class living as a writer of short fiction. It wasn't easy; you had to write well and type fast enough to melt the keys but a surprisingly large number of people managed to do it.

Nor were writers the only example of the new creative middle class. According to Rosy McHargue (reported by music historian Brad Kay) in 1925 there were two hundred thousand professional musicians in the United States. Some were just scraping by, but many were making a good living. (keep in mind that many restaurants, most clubs and all theaters had at least one musician on the payroll.) Likewise, the large number of newspapers and independent publishers meant lots of work for graphic artists.

I don't want to wax too nostalgic for this era. Sturgeon's Law held firmly in place: 95% of what was published was crap. But it was the market for crap that made the system work. It provided the freelance equivalent of paid training -- writers could start at least partially supporting themselves while learning their craft, and it mitigated some of the risk of going into the profession -- even if you turned out not to be good enough you could still manage food and shelter while you were failing.

It was also a remarkably graduated system, one that rewarded quality while making room for the aforementioned crap. The better the stories the better the market and the higher the acceptance rate. In 1935, Robert E. Howard made over $2,000 strictly through magazine sales. Later, as the paperback market grew, writers at the very top like Ray Bradbury or John O'Hara would also see their stories collected in book form.

Starting with Gold Medal Books, paperback originals became a force in 1950. This did cut into the magazine market and hastened the demise of the pulps but it made it easier than ever before to become a novelist. It was more difficult (though still possible) to make a living simply by selling short stories, but easier to make the transition to longer and more lucrative works.

It was, in short, a beautifully functioning market with an almost ideal compensation system for a freelance based industry. It produced some exceptionally high quality products that have generated billions of dollars and continue to generate them in resales and adaptations (not to mention imitations and unlicensed remakes). This includes pretty much every piece of genre fiction you can think written before 1970.

The foundation of that system, the short story submarket, is essentially dead and the economics and business models of the rest of the publishing industry has changed radically leading to the rise of marketing, the blockbuster mentality and what I like to call the Slim Jim conundrum.

Tune in next time.

Meta-Freakonomics

Joseph recently wrote a post referring to this post by Andrew Gelman (which was based on a series of posts by Kaiser Fung which check the veracity of various claims in Superfreakonomics -- welcome to the convoluted world of the blogosphere). Joseph uses Dr. Gelman's comments about the poor editing and fact-checking of the book to make a point about the disparity between the contribution editing makes and how little we reward it. He ought to know; I have frequently taken advantage of his good nature in this area, but at the risk of being ungrateful, I don't think the point applies here. Rather than being helpful, the kind of criticism Joseph and Gelman describe could only hurt Superfreakonomics.

Or put another way, if we approach this using the techniques and assumptions of the Freakonomics books, we can show that by foregoing a rigorous internal review process the authors were simply acting rationally.

Before we get to the actual argument, we need to address one more point in Joseph's post. Joseph says that providing critical read "is one of the most helpful things a colleague can do for you, yet one of the least rewarded." This statement is absolutely true for easily 99.9% of the books and manuscripts out there. It is not, however, true for the Freakonomics books. Between their prestige and the deep pockets of William Morrow, Levitt and Dubner could have gotten as many highly-qualified internal reviewers as they wanted, reviewers who would have been compensated with both an acknowledgment and a nice check. (Hell, they might even get to be in the movie.)

But if the cost and difficulty of putting together an all-star team of reviewers for Superfreakonomics would have been negligible, how about the benefits? Consider the example of its highly successful predecessor. Freakonomics was so badly vetted that two sections (including the book's centerpiece on abortion) were debunked almost immediately. The source material for the KKK section was so flawed that even Levitt and Dubner disavowed it.

These flaws could have been caught and addressed in the editing process but how would making those corrections help the authors? Do we have any reason to believe that questionable facts and sloppy reasoning cost Levitt and Dubner significant book sales (the book sold over four million copies)? That they endangered the authors' spot with the New York Times? Reduced in any way the pervasive influence the book holds over the next generation of economists? Where would Levitt and Dubner have benefited from a series of tough internal reviews?

Against these elusive benefits we have a number of not-so-hard-to-find costs. While the time and money required to spot flaws is relatively minor, the effort required to address those flaws can be substantial.

Let's look at some specifics. Kaiser Fung raises a number of questions about the statistics in the "sex" chapter (the one about female longevity is particularly damning) and I'm sure he overlooked some -- not because there was anything wrong with his critique but because finding and interpreting reliable data on a century of sex and prostitution is extraordinarily difficult. It involves measurement covert behavior that can be affected by zoning, police procedures, city politics, shifts in organized crime,and countless other factors. Furthermore these same factors can bias the collection of data in nasty and unpredictable ways.

Even if all of the sex chapter's underlying economics arguments were sound (which they are, as far as I know), there would still have been a very good chance that some reviewer might have pointed out flawed data, discredited studies, or turned up findings from more credible sources that undercut the main hypotheses. That doesn't mean that the chapter couldn't be saved -- a good team of researchers with enough time could probably find solid data to support the arguments (assuming, once again, that they were sound) but the final result would be a chapter that would look about the same to the vast majority of readers and external reviewers -- all cost, no benefit.

Worse yet, think about the section on the relative dangers of drunken driving vs. drunken walking. These cute little counter-intuitive analyses are the signature pieces of Levitt and Dubner (and were associated with Dr. Levitt before he formed the team). They are the foundation of the brand. Unfortunately, counter-intuitive analyses tend to be fragile creatures that don't fare that well under scrutiny (intuition has a pretty good track record).

The analysis of modes of drunken transportation would be one of the more fragile ones. Most competent internal reviewers would have had the same reaction that Ezra Klein had:

You can go on and on in this vein. It's terrifically shoddy statistical work. You'd get dinged for this in a college class. But it's in a book written by a celebrated economist and a leading journalist. Moreover, the topic isn't whether people prefer chocolate or vanilla, but whether people should drive drunk. It is shoddy statistical work, in other words, that allows people to conclude that respected authorities believe it is safer for them to drive home drunk than walk home drunk. It's shoddy statistical work that could literally kill somebody. That makes it more than bad statistics. It makes it irresponsible.

Let me be clear. I am not saying that Levitt and Dubner knew there were mistakes here. Quite the opposite. I'm saying they had a highly saleable manuscript ready to go which contained no errors that they knew of, and that any additional checking of the facts, the analyses or logic in the manuscript could only serve to make the book less saleable, to delay its publication or to put the authors in the ugly position of publishing something they knew to be wrong.

Gelman closes his post with this:

It's the nature of interesting-but-true facts that they're most interesting if true, and even more interesting if they're convincingly true.

Perhaps, but Levitt and Dubner have about four million reasons that say he's wrong.