How to REALLY lie with statistics

Darrell Huff had a witty explanation of how you can change the impression a graph gives by playing around with the scale and range of the y-axis, but even in a book called How to Lie with Statistics he never even considered changing just part of the scale of an axis. Of course, Huff was limited by the fact that his book was based on real statistical lies and screw-ups that he had found in the popular press as of the early Fifties. He also tried to limit himself to lies that were at least true in some technical sense. There was no way he could have anticipated Fox News.

The catch comes from Media Matters via Mark Thoma.


Here is the same graph with vertical lines for full comic effect.


Go ahead, measure them for yourself. It's fun.

In all fairness to Fox, the period from September '08 to March '09 did feel like a long time.

Bob Dylan, the Monkees and the flooded landscape analogy

Seth Godin's comments on Bob Dylan and the Monkees (which comes to us via Gelman via DeWitt via Tinkers via Evers via Chance) got me thinking about fitness landscapes. Here's the quote:

Let me first describe a distinction between the Monkees and Bob Dylan. Bob Dylan gets laughed or booed off the stage every ten years, whether he wants to or not. He got booed off the stage when he went electric and again when he went gospel, and most recently with his horrendous Christmas album. The Monkees never get booed off stage, because the Monkees play "Last Train to Clarksville" exactly the same way they did it 30 or 40 years ago. Here's the thing: Bob Dylan keeps selling out stadiums and no one goes to see the Monkees, because the Monkees aren't doing anything worth noticing. There are people who have succeeded who just keep playing the same song over and over again, whatever that is that they do.

Think of a musician's career as a landscape where creative decisions like repertory, genre, style, arrangements give the location and concert sales are the fitness function. (see here and here for previous posts on landscapes)

In Godin's example, the Monkees have stuck very close to a local maxima that has sank over the years (the sticking close part doesn't actually match reality all that well -- Mike Nesmith had a run of innovative and interesting projects in the early days of music video -- but for the sake of the post let's overlook that part). Any small to moderate change in repertory or arrangement or style would move them to a lower point on the landscape.

I think I may be stealing this from Stuart Kuafmann, but let's flesh out the metaphor a bit and add water. Our landscape dwellers can travel freely on dry land but they can only swim very short distances. Exactly how does this relate to our real life example? Remember that altitude in our landscape corresponds to ticket sales. In order to stay viable, ticket sales for a touring act have to stay above a certain level. If the sales fall below that level, the act loses bookings and can no longer cover its expenses. Of course, like any other business, the act can run at a loss for a while (swim) but that's obviously not a long term solution.

Godin suggest that a willingness to, in our analogy, move to another optima is the key to success. Dylan made the move and thrived. The Monkees stayed put and whithered. But how comparable were the two situations?

Dylan had a steady source of income from other artists covering his songs. In landscape terms, he was a good swimmer (of course, so was Nesmith who got a tiny check every time you used that little Liquid Paper brush). More importantly, Dylan didn't have that far to swim. He might not even have needed to get wet. At least a portion of Dylan's fan base were going to stay with him no matter where he went on the musical landscape and given his reputation (and phenomenal talent, though I'm trying to leave that out of the discussion), there was a maxima waiting for him at pretty much every genre and subgenre of popular music. Those moves might not have been as artistically or commercially successful as the ones he made but Dylan was going to remain viable no matter where he went.

What about about the Monkees? Musically they weren't a bad line-up. Dolenz was a veteran child actor, Jones was a Tony nominee for Oliver! and Tork and Nesmith were both accomplished musicians. Highly successful careers have certainly been built on less, but what did their career landscape look like? Compared to Dylan's collection of tightly-packed peaks, the Monkees had a lonely island surrounded by what looked like a large and empty ocean. The vast majority of their fan base was location specific. When they moved away from that location they hit deep water very quickly.

It is, of course, possible that the group could have focused on coming up with new songs and a new sound with the hope of finding a new audience. This is a dynamic landscape, and where the artist chooses to go is one of the factors that affects it. There might not be a concert market for the Monkees playing new grass or thrash metal now but that doesn't mean there won't be one in the future. Sometimes, by playing music no one wants to hear, you can create a demand for that music. To return to the landscape analogy, treading water in one spot can cause an island to rise up beneath you. It has been known to happen but it's probably not something you want to count on.

In the case of the Monkees, the water-treading strategy would be particularly risky since their reputation is likely to work against them if they try something radically new. This is probably why Nesmith chose to use his own much less well known name for the Grammy-winning Elephant Parts rather than trying to sell it as a Monkees project.

Which brings us back to Mr. Godin and the advice books he and other business gurus dump on the market every year. These books gush out at such a rate that there are actually companies that put out fifteen page versions so that executives can at least give the impression that they have read the latest releases. The Dylan/Monkees example is sadly representative. It takes one of business gurus' favorite truisms (take risks, i.e. move out of your comfort zone, i.e. they laughed at Henry Ford), bills it as a fundamental key to fabulous success (fabulous as in fabled as in obviously untrue) then backs it up with an irrelevant but impressive sounding example.

Godin is telling businesses to be like Bob Dylan and to make radical moves that may piss off your customers and invite scorn and mockery. The trouble is very few businesses are Dylan-at-Newport. The majority are the Monkees-at-the-state-fair. They have something they do reasonably well. If they stick close to their local maxima they can turn a decent profit and have a pretty good run. If they follow Mr. Godin's advice they will sink like a cinder block and never be heard from again.

Grants!

Professor in Training has a nice post on grant writing and the frustrations encountered in the process. I have to admit that the first point is the one that I find the hardest: with papers I can be relentless in getting them into journals. With grants, the limited number of submission options is very discouraging -- especially given my lack of success so far.

I know that I need to keep at it but it is one of the hardest things for me in my current role!

It is possible to have a simple question in a complex field

Kartik Athreya packs a lot of stupid into one PDF and though Mark Thoma does a good job of unpacking most of it, there are, inevitably, statements that could use additional rebuttal. Here's an example:

It is precisely from this low-level vantage point that I am totally puzzled by the willingness of many who fearlessly and breathlessly opine about economics, especially macro- economic policy. Deficits, short-term interest rate targets, sovereign debt are all chewed over with a level of self-assuredness that only someone who doesn’t know more could. The list of those exhibiting this zest also includes, in addition to those mentioned above, some who might know better. They are the patron saints of the “Macroeconomic Policy is Easy: Only Idiots Don’t Think So” movement: Paul Krugman and Brad Delong. Either of these men will assure their readers that it’s all really very simple (and may even be found in Keynes’ writings).

There's an obvious but important point that Dr. Athreya either manages to miss or avoid, namely the distinction between a simple field and a simple question. In almost all fields, even the most complex and challenging, there are simple, easily-answered questions.

Krugman and Delong frequently argue that some colleagues, policy-makers, journalists or politicians have reached the wrong conclusion about a simple point in macroeconomics. They sometimes also contend that some of these errors are partially the result of idealogical thinking or outside influence (both of which have occasionally been observed in economics).

Dr. Athreya is free to dispute Krugman and Delong's arguments either by showing that the question is not simple or that it is simple but they still got it wrong. If he could actually prove either he could really advance the debate. Instead he seems to argue that the existence of simple questions implies a simple discipline.

And for someone of Dr. Athreya's qualifications and accomplishments, that's kind of simple-minded.

A pre-post post on Bob Dylan, the Monkees and the perils of reading business books

Andrew Gelman has a post that refers to this passage from business guru Seth Godin:

Let me first describe a distinction between the Monkees and Bob Dylan. Bob Dylan gets laughed or booed off the stage every ten years, whether he wants to or not. He got booed off the stage when he went electric and again when he went gospel, and most recently with his horrendous Christmas album. The Monkees never get booed off stage, because the Monkees play "Last Train to Clarksville" exactly the same way they did it 30 or 40 years ago. Here's the thing: Bob Dylan keeps selling out stadiums and no one goes to see the Monkees, because the Monkees aren't doing anything worth noticing. There are people who have succeeded who just keep playing the same song over and over again, whatever that is that they do.

This got me thinking about adaptation and fitness landscapes and all sorts of other interesting topics but before I get into that I really should take a moment to point out that, like so many examples from business gurus, this is complete bullshit.

Big stadium shows are dominated by heritage acts singing decades-old songs. Almost none of these acts have done significant work recently; many of the biggest (the Stones, the Police) haven't even released a full studio album in the past decade. Dylan is an tremendous anomaly here. He's not quite unique (Neil Diamond also recorded some of his most critically and commercially successful material in the past decade), but he is spectacularly unrepresentative.

Just to sum it up:

1. Loads of acts from the Sixties,Seventies and Eighties are selling out concerts;

2. Two or three of them are trying something new;

3. All the rest are coasting by on greatest hits.

From this Godin implies that the key to success is not repeating yourself.

Some mathematical humor from Jeff Mallett's charming strip Frazz.

Social Norms

Justin Alexander has an interesting post on the case of Twixt; a media professor named David Myers decided to play in an online game. He decided to deliberately ignore the social conventions of the game (it being an online role-playing game) and focus entirely on what the rules would allow (even if doing so irritated, annoyed and frustrated his fellow players).

For me the key quote was:

Given the adaptive value of individual play in exploring and revealing system characteristics, the social pressures against this sort of play in CoH/V seem drastically and overly harsh, even unnatural.

This approach ignores that, in most societies there are two layers to behavioral constraint. One is the use of actual laws while the other is social norms of behavior that apply to interactions. These social norms act as an important break on places where the rules do not apply. Would we not have concerns with an person who used an innovative approach to the law in order to committ pre-meditated murder? Would we decide that this was an important legal innovation and encourage more such innovation? Or would we be seriously concerned about the actions of the person involved.

I think that this issue is central to understanding how to make innovation work well; ideally innovation needs to be both an improvement and be accepted by the potential users. While the Twixt case was rather extreme, it does have some important issues for Epidemiology. When we diagnose risk factors for disease, we also have to decide what the current social context accepts as a reasonable implementation of these findings. Some ideas, like standing desks, might take a while to gain social acceptance. Others, like smoking, may require decades long public health campaigns to finally reduce the levels of exposure, despite the potential health risks involved.

Innovation is a complicated business . . .

Advice on meeting people

Prof-like substance has a great post on how to break the ice in unfriendly departments. I rather suspect it would work just as well in speeding up the process of getting to know people in friendly departments as well. However, I am not a bench scientist and there is a sharp limit to the number of statistics books I can borrow.

Any ideas how how to generalize this advice to the epidemiology world?

Effect and cause: Hollywood edition

From New York Magazine:

How Toy Story 3 Scored: Even though the kiddies all clap their palms raw whenever that silly cyclops of a desk lamp hops out to squash the "I" in "Pixar," Disney still faced a conundrum: Those tykes who were in first grade when Toy Story first hit theaters in 1995 are now seniors in college.

However, instead of writing off the twentysomethings as too jaded to come, the studio targeted them, knowing that the Woody/Buzz bond was ageless. "Pixar went out of their way to ground the movie in the idea that the character [Andy who] you knew as a kid was now off to college," explained David Sameth, Disney's senior VP of marketing, "It’s only natural to pick up the same idea in marketing, and translate it into our terms." That meant courting the keg-stand-performing college kids directly: In the run-up to TS3's release, Disney sponsored a special "college cliff-hanger campaign," screening two-thirds of the finished film for undergrads at some 80 colleges nationwide, which kicked off the positive buzz from the Twitter set. The studio also crafted a viral YouTube campaign — fake period ads for one of the movie's new additions, Lots-o-Huggin' Bear — that is still being discovered.

Helped by a 100% favorable rating on RottenTomatoes.com, not only did those former grade schoolers come running back to make this Pixar's top-grossing opener ever (helped by 3D pricing), but so did everyone else: Its audience was 46% over the age of 25, and roughly equally distributed between males and females. Who went? With 4,000 plus theaters showing it, "Everybody," said Chuck Viane, Disney's distribution chief. But what about "franchise fatigue" and all that? Says Disney's Sameth, "People don't go to see franchises; they go to see movies. This is a great one."

I'll buy that last part. I haven't seen the film yet but based on the reviews, the talent and most importantly, the standard of work from Pixar, I will be surprised to see anything less than excellence.

I would also have been surprised to see the picture fail to bring in dump trucks of money. This kind of long-awaited sequel tends to do well (think of how many people went to see Harrison Ford hobble around in Crystal Skull or George Lucas burn off what remained of his legacy in Phantom Menace). Pixar films also tend to bring in big money (Up grossed nearly three quarters of a billion worldwide). Add to that the fact that after three films built around gifted but non-bankable character actors (Oswald, Asner, Plummer) and uncomfortable themes (rats preparing food; the destruction of the environment; old age, loneliness, and even miscarriage), Pixar played this one safe with a big star in the lead and familiar thematic territory. It is probably the least adventurous film from the studio since, well, Toy Story 2.

Given all this and the significant traditional media marketing, there is no evidence here that targeting college students brought in anyone who wouldn't have seen the film anyway. This type of alternative, targeted marketing can be highly effective for movies and other products you might not have otherwise heard of (Kick-Ass, Defendor, OSS 117 would all be reasonable choices). The techniques are much less effective for well-publicized films and they aren't scalable; a good twitter/viral video campaign can push a small film into wider release which can take you from a gross of five million to ten or twenty million, but when you're talking about opening in thousands of theatres and having to gross two hundred million just to break even, this kind of non-traditional marketing is usually a waste of time.

But the entertainment industry loves to tell itself these stories: marketing to twenty-somethings helped make Toy Story 3 a hit; the Hangover shows that people want gross out gags and Apatow-style humor; Electra, Catwoman and Charlie's Angels II all bombed because they were female superhero movies with dark themes. None of these stories stand up to any scrutiny. All can be replaced by more credible explanations (usually starting with the quality of the script). But the industry still embraces these unbelievable accounts because of what they say about replication and risk.

William Goldman famously observed that when it comes to how well a movie is going to do "Nobody knows anything." That's not quite true. There is an optimal strategy: start with a strong script; keep people (particularly studio executives) away from it as much as possible; hire a competent director and a cast of good actors who fit their roles; provide an adequate marketing campaign that gives the potential audience an accurate impression of the film.

As straightforward as this may seem, this strategy gives little comfort to people in the industry for two reasons.

First, this is difficult to replicate on a large scale. There are writers who can turn out strong scripts (Goldman being the most obvious example) but they are hard to find and can take a long time to cultivate. To put together a team comparable to what Pixar has is a monumental task.

Second (and this is where Goldman's observation really kicks in), there is a great deal of unpredictability in the system. Sometimes film-makers will do everything right and the film will just fail to gel artistically. Other times a film will come out perfect but for some reason won't get the reception it deserves.

This need to find a comforting explanation for success and failure is not limited to Hollywood. Many if not most companies spend a great deal of time retroactively assigning causes to major successes and failures. There is usually little empirical evidence at work and quite a bit of politics and score settling, but even the most dubious of explanations have a way of making it into the official unofficial history.

For years (perhaps even to this very day), executives at McDonald's would tell you with absolute certainly that the deluxe line failed not because it was based on over-hyped, under-impressive menu items but because the ads showed Ronald playing golf.

Credit Hours

Dead Dad has an intersting take on congress trying to make the credit hour a standard measure to qualify for student aid. The real issue here is not the productivity paradox that he defines but, rather, the issue of creating clear metrics in higher education. The original impetus for the move was a poor decision on the part of an accrediting body; but the truth is that, so long as there is so much money at stake, there will be entities that try and exploit the rules.

Switching to an outcomes based approach is a tempting idea until you realize how hard it is to measure outcomes in education clearly. Do we want to create perverse incentives to select only strong students because they are the most profitable group?

It's a really tough problem!

Co-authored work

This is a major tangent from the post that inspired it, but going radically off topic is on of the tropes of the internet. In the post plus comments, a post-doctoral fellow expresses some frustrations about troubles getting a co-authored piece of work to publication.

One of the hardest things in academia is having the discipline to let the leaders of co-authored work (first and senior) author focus on the process of moving a paper to final publication. It's hard to stand back when you are excited by a piece of work and it is easy to be frustrated. But it is also the only way I can stay sane is to trust my co-authors and to be available to provide help as needed.

But I admit to preferring to be in the thick of things . . . :-)

Dying is easy; comedy is hard -- journalism edition

"When they see us coming, the birdies all try an' hide,
But they still go for peanuts when coated with cyanide."

It started with poisoning pigeons in the park.

Maybe I should be more specific.

It started a few weeks ago when I heard the song "Poisoning Pigeons in the Park" as part of a Tom Lehrer* tribute on Prairie Home Companion. Impressed by the wit of the lyrics (the man rhymed cyanide, for God's sake) I decided to see what a professional song writer would think about Lehrer.

My go-to guy in these matters is Brad Kay, songwriter, music historian and former mystic knight. Brad can be a tough critic, particularly if your name isn't something like Gershwin, Waller or Ellington, but if I was expecting him to be dismissive of someone who was just a comic songwriter I was in for a surprise.

Brad had literally nothing but praise for Tom Lehrer. He talked about how graceful and apt Lehrer's choice of words was and about profoundly he understood each of the genres he worked on. This led to discussions of Spike Jones (who was an accomplished and successful percussionist before he started using gunshots and noisemakers in his music) and P.D.Q. Bach (who as Peter Schickele has composed a large number of well-regarded symphonies, musicals, and film scores). All of this suggested that the first requirement for creating comic music of more than passing interest was being a good musician.

It's probably not surprising that the general public has trouble taking the creators of comedy seriously, but their peers have no such difficulty. P.G. Wodehouse counted George Orwell among his fans. Any number of dancers and acrobats have commented on the grace and athleticism of Charlie Chaplin and Buster Keaton (Keaton was particularly remarkable having mastered that most difficult of athletic feats, the controlled fall, by age three). As for acting, it's almost a truism that comic actors can easily do drama while dramatic actors often struggle with comedy.

So if the best musicians, writers, dancers and actors can be found doing great comedy, how about journalists?

The following clip is a thoroughly typical segment of the Daily Show. Take a look and consider it from a journalistic standpoint. Look at how clear and concise Stewart is. Check out how important but underreported facts are introduced and how everything is placed in a relevant context. I wonder how many hours of cable news you'd have to watch to meet the standards of just another Daily Show?

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
Indecision 2010 - Primary Victory for Women
www.thedailyshow.com
Daily Show Full Episodes Political Humor Tea Party

*Arguably the world's coolest mathematics professor.

More on impacts of Research

From FemaleScienceProfessor comes a really nice comment on publishing in important journals:

I have absolutely no problem publishing in a disciplinary journal with impact factor of 2-4. These are excellent journals, read by all active researchers in my field. It is bizarre to compare them unfavorably with Nature and Science, as if papers in a journal with an impact factor of 3 are hardly worth reading, much less writing.

This is entirely correct. Several of the key journals that I real all of the time are in this category: Pharmacoepidemiology and Drug Safety and Statistics in Medicine, to name a couple of obvious ones.

I think, to some extent, this is the dark side of a focus on impact. A finding that is of interest to a broad audience may not necessarily be a good measure of the importance of a finding to a sub-field. There are a lot of papers, to use an example, that say "further work in this direction looks challenging". This is important information to let other researchers know but it's likely to be heavily cited.

Now, there is a case to be made for the volume of research publications that are made being harder and harder to follow with time. But this can also be a benefit -- many clusters of researchers working in different directions may be able to better find optimal solutions that would otherwise be overlooked.

It is a complex issue but I'm definitely in favor of discipline specific publications and some of my favorite papers are in this type of niche.

No subtle and profound scientific insights from Olivia Judson this week

Just a sad and straightforward account of people contemplating an act of tragic stupidity.

Socio-economic Models

Mark has a great comment on issues with interpreting what a dollar is worth. This idea is worth considering in Epidemiological studies as well. When trying to model socio-economic status, it can be tempting to treat income as a linear variable. Even worse, it can be tempting not to account for differences in costs of living. This reluctance to try and account for cost of living is an understandable temptation as the data on cost of living is extremely hard to generate and very context specific. How do you handle Manhattan (where a large house is unlikely to be even available, let alone affordable) when you compare it Duluth, Minnesota. Economics have some clever tricks, but they replace one source of error (incorrect standardization) with another one (modeling errors in the cost of living models).

But it is worth remembering that just because it is a clean number doesn't mean that you should just introduce it into the model "as is". Think long then model!