Robert Samuelson would not make a good statistician

Robert Samuelson is taking considerable heat for this column in the Washington Post complaining about the way we measure poverty. Dean Baker and Mark Thoma posted detailed and highly critical responses that listed several problems with Samuelson's argument. Both of them, however, skipped over at least one serious statistical flaw in the column.

Here's the quote from Samuelson:

Second, the poor's material well-being has improved. The official poverty measure obscures this by counting only pre-tax cash income and ignoring other sources of support. These include the earned-income tax credit (a rebate to low-income workers), food stamps, health insurance (Medicaid), and housing and energy subsidies. Spending by poor households from all sources may be double their reported income, reports a study by Nicholas Eberstadt of the American Enterprise Institute. Although many poor live hand-to-mouth, they've participated in rising living standards. In 2005, 91 percent had microwaves, 79 percent air conditioning and 48 percent cellphones.

The fallacy here is closely related to the phenomena of the wrong-way coefficient. You fit a model and you see a statistically significant variable with the wrong sign. For a fairly silly example, you build a model predicting how long it takes travellers to get from New York City to DC and you find that the indicator for being searched by a uniformed officer has a negative coefficient which would suggest that being searched somehow shortens your travel time. The explanation for this counterintuitive result is that there's a relationship between this variable and one or more of the other variables in your model. In this case there's a strong correlation between being searched and flying vs. driving.

For people living in residences with functioning kitchens, good ventilation and a land line, getting a microwave, an air conditioner and a prepaid cellphone clearly represents an increase in well being. If, however, there is an inverse relationship among the poor between having a stove/having a microwave, or ventilation/AC or land line/cell, then the high incidence rates could easily indicate a lower standard of living.

For an example of how not having a stove could make having a microwave more likely, check out this story from NPR:

So many immigrants, homeless people and others of limited means living in single-room occupancies (SROs) have no kitchens, no legal or official place to cook. To get a hot meal, or eat traditional foods from the countries they've left behind, they have to sneak a kind of kitchen into their places. Crock pots, hot plates, microwaves and toaster ovens hidden under the bed. And now, the latest and safest appliance, the appliance that comes in so many colors it looks like a modern piece of furniture: the George Foreman Grill. It is, quite literally, a hidden kitchen.

For me, a George Foreman grill would be a luxury purchase, but not having one doesn't mean I'm worse off than the next guy I see pushing a shopping cart with all of his belongings down the street.

Network TV -- now entering their fourth decade on death's door

Brian over at Ultrasonic Remote sends this latest obituary for network television based on this:

YouTube’s viewership now exceeds that of all three networks combined during their “primetime” evening time slot, with more than 2 billion views per day, Google announced Sunday.

There are a lot of statistical flaws in the article but the one that jumped out at me was the bit about "all three networks." This is a particularly embarrassing variation on an old fallacy. Articles on the decline of network viewership have often failed to take new networks into account, for example comparing CBS/ABC/NBC viewership in 1980 and 1995 then concluding that the difference represented people going to cable.

You can see how someone would make that mistake. It seems to be an apples-to-apples comparison even though it's not.

But in a discussion limited strictly to 2010 viewership, talking about "all three networks" is just weird.

The economics of rock and roll part III -- Gimme (tax) Shelter

From APR's Marketplace:

The image may be rebellious and unkempt, but the reality is that the Rolling Stones are very smart when it comes to money. On worldwide earnings of $150 million last year they paid just 1.5 percent in tax.

The details have emerged under Dutch law. The band's financial advisers apparently have offices in a Dutch tax haven. The lightness of the Stones' tax burden is not a surprise, says former rock musician Neil MacCormick:

NEIL MACCORMICK: They're a big machine. They've probably learned the hard way, and learned on the road how to protect their money.

He says world tours are usually organized according to tax rules, with a band rarely appearing for longer than 30 days in each country.

The economics of rock and roll part II -- quality control and brown M&Ms

This one has been passed around a lot but it's a nice example of a clever solution to a business problem and it bears repeating.

From Snopes:

The legendary "no brown M&Ms" contract clause was indeed real, but the purported motivation for it was not. The M&Ms provision was included in Van Halen's contracts not as an act of caprice, but because it served a practical purpose: to provide an easy way of determining whether the technical specifications of the contract had been thoroughly read (and complied with). As Van Halen lead singer David Lee Roth explained in his autobiography:

Van Halen was the first band to take huge productions into tertiary, third-level markets. We'd pull up with nine eighteen-wheeler trucks, full of gear, where the standard was three trucks, max. And there were many, many technical errors — whether it was the girders couldn't support the weight, or the flooring would sink in, or the doors weren't big enough to move the gear through.

The contract rider read like a version of the Chinese Yellow Pages because there was so much equipment, and so many human beings to make it function. So just as a little test, in the technical aspect of the rider, it would say "Article 148: There will be fifteen amperage voltage sockets at twenty-foot spaces, evenly, providing nineteen amperes . . ." This kind of thing. And article number 126, in the middle of nowhere, was: "There will be no brown M&M's in the backstage area, upon pain of forfeiture of the show, with full compensation."

So, when I would walk backstage, if I saw a brown M&M in that bowl . . . well, line-check the entire production. Guaranteed you're going to arrive at a technical error. They didn't read the contract. Guaranteed you'd run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.

The economics of rock and roll

Tyler Cowen passes on the following sharp observation from Mick Jagger on the upheavals in the music industry:

...people only made money out of records for a very, very small time. When The Rolling Stones started out, we didn’t make any money out of records because record companies wouldn’t pay you! They didn’t pay anyone!

Then, there was a small period from 1970 to 1997, where people did get paid, and they got paid very handsomely and everyone made money. But now that period has gone.

So if you look at the history of recorded music from 1900 to now, there was a 25 year period where artists did very well, but the rest of the time they didn’t.

Jagger, of course, studied economics at LSE and is known to be a fan of Hayek.

I ran this by the music historian Brad Kay, who confirmed that Jagger had his facts right. Thanks to Tyler for posting this and more thanks to Gut(?) who commented:

He is also the primary datapoint for the observation that economists do it with models.

Elegance and Economics -- a choice comment from Paul Krugman

From "How did Economists Get It So Wrong?" (September 2nd, 2009):

Few economists saw our current crisis coming, but this predictive failure was the least of the field’s problems. More important was the profession’s blindness to the very possibility of catastrophic failures in a market economy. During the golden years, financial economists came to believe that markets were inherently stable — indeed, that stocks and other assets were always priced just right. There was nothing in the prevailing models suggesting the possibility of the kind of collapse that happened last year. Meanwhile, macroeconomists were divided in their views. But the main division was between those who insisted that free-market economies never go astray and those who believed that economies may stray now and then but that any major deviations from the path of prosperity could and would be corrected by the all-powerful Fed. Neither side was prepared to cope with an economy that went off the rails despite the Fed’s best efforts.

And in the wake of the crisis, the fault lines in the economics profession have yawned wider than ever. Lucas says the Obama administration’s stimulus plans are “schlock economics,” and his Chicago colleague John Cochrane says they’re based on discredited “fairy tales.” In response, Brad DeLong of the University of California, Berkeley, writes of the “intellectual collapse” of the Chicago School, and I myself have written that comments from Chicago economists are the product of a Dark Age of macroeconomics in which hard-won knowledge has been forgotten.

What happened to the economics profession? And where does it go from here?

As I see it, the economics profession went astray because economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth. Until the Great Depression, most economists clung to a vision of capitalism as a perfect or nearly perfect system. That vision wasn’t sustainable in the face of mass unemployment, but as memories of the Depression faded, economists fell back in love with the old, idealized vision of an economy in which rational individuals interact in perfect markets, this time gussied up with fancy equations. The renewed romance with the idealized market was, to be sure, partly a response to shifting political winds, partly a response to financial incentives. But while sabbaticals at the Hoover Institution and job opportunities on Wall Street are nothing to sneeze at, the central cause of the profession’s failure was the desire for an all-encompassing, intellectually elegant approach that also gave economists a chance to show off their mathematical prowess.

Distributed computing

In medical research, data privacy is paramount. So it is unlikely that we'll be taking advantage of things like cloud computing anytime soon.

However, even if I could, I doubt that I would in the absence of brutally complete back-ups (which would rather eliminate any space savings). Because the world is broken up into two groups of people: those who have lost data due to crashes and those who are yet to do so. When you are in the former group, it seems the height of insanity not to be as a careful as possible about keeping data safe.

Heck, I dislike how the SAS program editor does not have an autosave feature. I've lost code because I walked away from the computer for a meeting and windows decided that was an ideal time for an auto-update. In that sense, the static environment of using the computer as a tool is at war with the dynamic vision of computer developers.

I wonder if, at some point, we'll branch off into two streams? Computers are cheap enough that you no longer need one platform for everything and I mostly use the internet at work for downloading articles (a function that slowly updated libraries could pretty much handle).

It is an interesting thought.

"How is model-based macroeconomic forecasting possible?"

There's an interesting discussion of economic forecasting over at Worthwhile Canadian Initiative. The post and the responses raise some points I've been wondering about myself. It's good to see people inside the field addressing them.

"Baby Steps to New Life-Forms"

The latest from Olivia Judson.

What Auteur Theory and Freshwater Economics have in common

(the first draft is the dominant genre of the internet. Between the roughness of this essay and my extensive ignorance of criticism and economics, I'm sure there is plenty of room for improvement here. If any readers have suggestions for taking this to the next level please let me know.)

(you might want to read this New York Times piece by Paul Krugman before going on -- it's the best primer I know of for this debate.)

We'll define freshwater economics as the theory that economic behavior (and perhaps most non-economic behavior) can be explained using the concepts of rational actors and efficient markets and auteur theory as the idea that most films (particularly great films) represent the artistic vision of a single author (almost always the director) and the best way to approach one of those films is through the body of work of its author. Both of these definitions are oversimplified and a bit unfair but they will get the discussion started.

At first first glance, these theories don't seem to have much in common, but as we step back and look at them in general terms, fundamental similarities start to emerge in their styles, their ecological niches and in the way they've been received.

Compared to their nearest neighbors, film criticism and economics (particularly macroeconomics) are both difficult, messy fields. Films are collaborative efforts where individual contributions defy attribution and creative decisions often can't be distinguished from accidents of filming. Worse yet, most films are the product of large corporations which means that dozens of VPs and executives might have played a role (sometimes an appallingly large one) in determining what got to the screen.

Economists face a comparably daunting task. Unlike researchers in the hard sciences, they have to deal with messiness of human behavior. Unlike psychologists, microeconomists have few opportunities to perform randomized trials and macroeconomists have none whatsoever. Finally, unlike any other researchers in any other field, economists face a massive problem with deliberate feedback. It is true that subjects in psychological and sociological studies might be aware of and influenced by the results of previous studies but in economics, most of the major players are consciously modifying their behavior based on economic research. It is as if the white mice got together before every experiment and did a literature search. ("Well, there's our problem. We should have been pulling the black lever.")

Faced with all this confusion, film scholars and economists (at least, macroeconomists) both reached the same inevitable conclusion: they would have to rely on broader, stronger assumptions than those colleagues in adjacent fields were using. This does not apply simply to auteurists and freshwater economists. Anyone who does any work in these fields will have to start with some sweeping and unprovable statements about how the world works. Auteurists and freshwater economists just took this idea to its logical conclusion and built their work on the simplest and most elegant assumptions possible, like Euclid demonstrating every aspect of shape and measure using only five little postulates.

(Except, of course, Euclid didn't. His set of postulates didn't actually support his conclusions. The world would have to wait for Hilbert to come up with a set that did. The question of whether economists need a Hilbert will have to wait for another day.)

Given that we have two similar responses to two similar situations, it is not all that surprising to see that both schools of thought have followed similar paths and have come to dominate their respective fields. I don't think that anyone would argue that any institution has had more impact on economics than the Chicago school over the past fifty years and I doubt you could find a theory of film that comes close to the impact of auteurism over the same period.

This dominance was achieved despite serious criticisms and counter-examples. When the writer William Goldman (Butch Cassidy and the Sundance Kid, Princess Bride, many, many, many others) heard about auteur theory, his reaction was "What's the punchline?" The sentiment was echoed by many writers who pointed out that many of the elements that the critics discussed were determined, explicitly or implicitly in the script. On related grounds, others pointed out how many of the creative decisions were made in preproduction often before the director was hired (John Huston said that a film was mostly finished once you had the cast and the script). Others talked about films that were "saved in the editing room," a common Hollywood expression for films that are radically changed for the better in post-production, usually after having been taken away from the director. Many (including Goldman) argued that films were the sum of many individual contributions and that changing any of them would result in a different movie.

Critics of classical economics question the realism of the school's postulates. They suggest that the proposed 'homo economicus' would have to be a cross between a lightning calculator and a high-functioning psychic. They point to findings from behavioral economics that show individuals failing to act according to neoclassical principles and historical cases where neoclassical models failed to predict economic events.

Both schools of thought partially address these complaints by arguing that their critics are trying to apply their ideas in cases where the necessary conditions don't hold. For auteurists, conditions included technical competence, recognizable style and a sufficient body of work. For freshwater economists, conditions included symmetry of information, a sufficient pool of buyers and sellers, a lack of externalities and freedom from government influence. These conditions did not refute the criticisms but they did provide defensible positions.

There is nothing unusual, let alone improper about proponents of a theory laying out conditions that have to be met before their concepts can be applied. (I could have written essential the same paragraph about Keynesians or deconstructionists.) What makes this notable is the disconnect between this approach and the way lay people use these ideas.

The dominance of auteurism and the Chicago School is, if anything, greater when you venture outside of academia. Most financial journalists, pundits and politicians take the power of market forces as a given and the vast majority of movie reviewers routinely assume that the director is the author of the film they just saw, but in both these cases with very few exceptions, the lay people using these theories have no idea that the conditions of the previous paragraphs even exist.

The problem with auteurism is compounded by the fact that most reviewers have no idea what a director actually does. This was certainly not true of the original French critics who popularized the theory (who were, themselves, directors) or of its primary American proponent, Andrew Sarris, (who went to great pains to discuss exactly and also set out the definitive list of the conditions I referred to).

Today most reviews will use the possessive form of the director's name then proceed to discuss everything about the film but the direction. The strange result of all this is that directors are both the most overrated and under-appreciated of movie makers. They are given credit for the work of everyone else while their own contribution is generally ignored.*

Obviously, the stakes are higher for economics but the disconnect is just as big. Open up any op-ed page or tune in any news conference and you are likely to find someone using freshwater arguments in situations where any serious freshwater economist would tell you they don't apply. For example, it is easy to pundits and politicians who believe we should let the market forces handle global warming instead of having a carbon tax, despite the pro-tax position of economists like Laffer, Cowen and Mankiw. It isn't that these laymen are consciously disagreeing with these experts; they simply don't know that using taxes to address externalities is a fundamental part of the philosophy they think they are espousing.

Finally, both schools had clear winners and have been aggressively promoted by those winners. Directors were the big winners in auteur theory; they gained power and prestige which in an industry that knows how to reward those attributes. It may not have been entirely a coincidence that the original auteurist critics had their careers as directors enhanced by the rise of the theory.

In economics, there is no question that the rise of freshwater ideas and approaches have been advanced considerably by institutions like the American Enterprise Institute and the Heritage Foundation, nor is there any question that much of the funding for these institutions came from companies that directly benefited from the dominance of freshwater economics.

Do these schools deserve their positions of dominance? That's a question for people above my pay grade. I'm just pointing out that widely separated disciplines can be surprisingly similar when you take things to a high enough level.


* For a view of how little influence some directors have on actors' performances check out these comments by Robert Mitchum (cutter in this context means film editor).

note: The Paul Krugman link at the top was added 5/31/10.

Journal Reviewing

How often do you review?

In the latest early releases from the American Journaol of Epidemiology, the editors discuss the concern of authors who submit heavily but do not accept requests to review articles. This situation is a classic case of the "Tragedy of the Commons"; the credit for reviewing is small (and typically anoymous). If nobody reviewed then the peer-review system would break down. However, the cost of any one person removing themselves from the reviewer pool is small compared to the benefit (for that person) in being able to focus on their own research projects.

Even worse, it is hard to know if the person who refuses at journal X might be a common reviewer at journal Y. Is the person who refuses refusing because they already do a lot of reviewing for another journal? Heck, I have reviewed for the American Journal of Epidemiology and I've never had a paper accepted there!

Female Science Professor once assembled a list of why people might see reviewing as a rewarding experience. It is an interesting list but maybe not completely convincing that the benefits outweigh the costs.

On the other hand, it is somewhat insirational that this system works despite everything that is working against it. Maybe science still has a critical mass of idealists left?

It's rather a nice thought.

Recursion humor

I think Mr. Gardner would have appreciated this. From Stanley Stories:

One less giant

Martin Gardner
(October 21, 1914 – May 22, 2010)

From Mark Thoma, two links and a damned good observation

First Thoma links to this:

How to prevent huge teacher layoffs, by Christina D. Romer, Commentary, Washington Post: The emergency spending bill before the House would address the education crisis facing communities across America -- and the jobs of hundreds of thousands of teachers are at stake. Because ... state and local budgets are stressed to the breaking point..., hundreds of thousands of public school teachers are likely to be laid off over the next few months. As many as one out of every 15 teachers could receive a pink slip this summer...

Then this:

Bill on jobless benefits, state financial help scaled back, by Lori Montgomery, Washington Post: Under fire from rank-and-file Democrats worried about the soaring national debt, congressional leaders reached a tentative agreement Wednesday to scale back a package that would have devoted nearly $200 billion to jobless benefits and other economic provisions...

Which he follows with the following observation:

The deficit hawks generally talk about the fate of our children when making the case to reduce the deficit, but at a time like now when the recession is pressuring school budgets, how are kids helped by reducing their educational opportunities?

It used to be fun reading Molly Ivins skilfully mocking hypocritical politicians trotting out the "What about the children?" line. Now she's gone. They're still here.

And I'm depressed as hell.

DIY-U

In a recent post, Dead Dad wrote about DIY-U. And he's right -- most academic training (especially in areas like Epidemiology) benefits greatly from an institutional framework. But there is another angle that I think is worth thinking about -- school should not be the only place from which you derive learning. There are huge advantages to being well read and interested in a wide range of subjects. It's not an optimal strategy to take a degree in every area you have any kind of interest in.

So let us try and take the good part of this bad idea -- lifelong learning is something that it is important to begin early and never really stop doing!

Pudding wrestling, the Antichrist and the breakdown of a winnowing system

Brian (who blogs on television and society at Ultrasonic Remote) sent the following example of winnowing processes gone awry by the always sharp Gail Collins. Collins seems to doubt that the GOP primaries are selecting their strongest candidates:

Pretend you’re the Republican leadership in a smallish state with an open United States Senate seat. The opposition is running a popular, longtime officeholder whose sense of inevitability was shaken by recent revelations that he had referred to himself as a Vietnam War veteran when he isn’t one.

Your own options are:

A) A well regarded former congressman who is a decorated Vietnam War veteran.

B) A political novice who made her fortune building up an entertainment business that specialized in blood, seminaked women and scripted subplots featuring rape, adultery and familial violence. In which the candidate, her husband and children played themselves. Also, the family yacht is named Sexy Bitch.

Well, obviously, you go for the yacht owner.

Yes, this week the Connecticut Republican Party chose Linda McMahon, the former C.E.O. of World Wrestling Entertainment, to be their Senate candidate. Her main opponent, the former Representative Rob Simmons, packed up his war medals and went home.

“You can’t argue with arithmetic,” he told The New London Day.

The math in question is $50 million, the amount McMahon claimed she was prepared to spend on her campaign. Connecticut has just under two million registered voters, so maybe she’ll just invite everybody in the state to a nice dinner at Red Lobster.

So far this season, the Republicans have offered two new models of their future. One is the Tea Party vision, in which outsiders full of spirit and excitement overthrow the old order. In North Carolina, there was so much spirit and excitement that voters gave the top spot in a Congressional primary to a former drug addict who, according to court documents, once referred to the United States government as the Antichrist and claimed to have personally located the Ark of the Covenant.

...

“One good thing has come from her run: Vince McMahon putting out an edict that there will no longer be any cutting of your foreheads with razor blades,” said Superstar Billy Graham, a retired wrestler who contracted hepatitis from a bloody competitor. “He has actually stopped wrestlers from cutting their heads with razor blades. This is a big deal!”

Still going better than California.

Steve Poizner needs to get out more

As reported on This American Life (get a download here. It's well worth the money), back in 2002, recently retired multi-millioniare Steve Poizner volunteered to teach a class in east San Jose. A few years later he wrote about that year in the book Mount Pleasant: My Journey from Creating a Billion-Dollar Company to Teaching at a Struggling Public High School. It's an interesting account (though not in any of the ways Poizner intended).

One detail struck me as particularly funny as I listened to the story while driving through LA. Here's the excerpt as Poizner read it on the show:

Several miles and a couple of highways later I took the Capital Expressway exit and drove into what felt like another planet. Signs advertising janitorial supply stores and taquerías. Exhaust hung over 10 lanes of inner city traffic; yellowing, weedy gardens fronted many of the homes, as did driveways marred by large oil spots or broken down cars.

Taquerias are generally small restaurants that serve mainly tacos and burritos. After hamburger joints, they are probably the most common restaurants in California. The thought that a California resident who wasn't looking for a place to eat would even notice them is odd. Listing them as an exotic sign of urban decay is downright bizarre.

the plural of anecdote is not data

Andrew Gelman has a great post on how statisticans do not always apply the lessons of statistics to their everyday lives. And it is true -- a lot of what we do in our lives is from custom and tradition that has never undergone rigorous evalaution. I'd be curious to see how teaching would change if we tested pre- and post-class knowledge (and had a control group who did not take the class). One suspects that the results might be humbling . . .

This reminds me of Miquel Hernan arguing in the journal Epidemiology (sadly, gated) that epidemiologists should be skeptical of journal impact factors because of our training. Unfortunately, despite this skepticism, it is rare for an active researcher not to have a good idea of the ranking of various journals (often based off of the impact factor). I think that staticisticans are better at disregarding impact factors but that's not a universal rule.

But I wonder if we could improve a lot of everyday tasks with a slightly tigher focus on statistical and/or epidemiological reasoning?

More great reporting from This American Life

This is either the best piece of California political reporting I've come across recently or the best piece of education reporting. Either way it's worth the ninety-nine cents.

"Study: Excluding Cellphones Introduces Statistically Significant Bias in Polls"

Nate Silver has a timely and insightful post on the state of political polling.

What happened before never happened before -- a quick aside on the equity premium debate

I've been enjoying this online discussion (particularly the contributions of Felix Salmon) and I'm sure Joseph and I will have more posts on the subject but first I have something I have to get off my chest:

No one can possibly know what's going on here! We can get some smart people making good guesses about long term stock performance, but these guesses are based on data from a century's worth of secular upheavals. A list that includes the Great Depression, two world wars, a post-war reconstruction, the cold war, China becoming a major player, boomers entering the market, boomers leaving the market and huge changes in regulation, technology and business practices.

What's happening now never happened before. What happened before never happened before. What happened before what happened before never happened before. We have no precedents. People are recommending forty-year investment strategies using models based on data from markets that haven't gone twenty years without a major secular change.
.
There. I've got it out of my system. Go on with what you were doing

John C. Bogle

We've been speaking about investment and stock market returns recently as an example of how forecasting is difficult. But I would be remiss not to mention Vanguard and it's very articulate founder. I've just finished reading his latest book (Enough) and it was very worth a couple of hours of time. A very thoughtful reflection on character and ethics.

I'm also hoping to emulate him in another way -- he is 91 years old and still active in what he loves. Here is hoping I'm still doing Epidemiology at the same age!

Publishing

Candid Engineer points out something that I think all of us find as a challenge -- actually getting work out in the form of papers. I like to solve puzzles and understand the world around me. It's what drives me as a scientist.

On the other hand, it's a minority of results that are fun to write up and the publication process is always painful. So I, too, have to watch the tendency to learn the results and then move on. This is specially true if you take on a high risk project and the results are a bit disappointing.

My most recently submitted paper was analyzed 2 years ago and it took a lot of effort to clean it up until it could be submitted. Not a fun process.

But sticking with it is crucial. It's the only way to really let others know what you have done and really push science forward!

The golden age of financial journalism

That sounds sarcastic but it's not. We really are blessed with an amazing crop of writers who can discuss business and economics with insight, encyclopedic knowledge and surprisingly good prose.

Here a couple of the best get us up to speed on financial reform:

Preventative Measures

Stop Complaining, Wall Street

The perils of forecasting

Mark points out, in a comment on my post on financial literacy, the issue of the equity premium as being discussed by Felix Salmon. I think that this point is really interesting and deserves more attention than a breif comment.

This article from the Economist is another great discussion of the complexities of making this claculation. The author lays out the basic problem with calculating an equity premium (or lack thereof) using current data (i.e. recent rates of return). One does not want to price in the current economic crisis (unless one thinks that these crises will be more common in the future) but neither does one want to calculate an equity premium that ignores either key events or actual real rates of return.

When you add in the issue of limited data (we only have a couple of hundred years of annual stock returns for the United States and only about 60-80 years are really relevant to the current market) and the risk of a secular shift (what if one or more financial innovations has fundementally changed the nature of the market) then these complications are almost enough to make the problem intractable. One might argue that these factors need to balance out in the long run (if equities don't pay a premium for risk then people will stop holding equities). But it is remarkable how long the long run can be; I think John Maynard Keynes said it best with:

The long run is a misleading guide to current affairs. In the long run we are all dead. Economists set themselves too easy, too useless a task if in tempestuous seasons they can only tell us that when the storm is past the ocean is flat again.

I think that this problem is true for epidemiological forecasts as well. The forecast of influenza rates (as a recent example) depends critically on the assumption that the current strains of influenze are not fundementally different than past strains. Often this assumption is reasonable but it can miss the most important changes (like the arrival of a new and more lethal version of the virus).

Whether it is disease rates or stock markets, it is not a simple matter to use the past as a guide to the future. There is no doubt that forecasting is hard but it's also true that it is important to do it as well as possible. If I ever figure out the trick I will be sure to share it!

TPM: still clueless in CA

Talking Points Memo has been doing a superb job covering the races in Kentucky and Pennsylvania. Josh Marshall's posts on Rand Paul have been particularly insightful. You really ought to take a look.

That has nothing to do with the main point of this post but I feel guilty hammering away at TPM and I'm afraid it's time to pick up the old nine-pounder again and start swinging at this:

Whitman's support has fallen in large part due to Democratic attacks over her connections to Goldman Sachs -- the Dems would prefer to face Poizner in the fall. There may also have been a backlash against her big personal spending on the race, which has reached $68 million so far, and a tightly controlled media operation in which she has avoided directly answering questions from reporters about the issues -- a fact that is frequently noted in media reports.

As statisticians, we are constantly being asked why something happened. We don't like the question (the subject of causality makes us nervous) but it's not something we can avoid so we approach it as rationally as we can. We look at correlations, of course, but we also look at timing, magnitudes, precedents. We consider the implications of the different hypotheses (for example, if bad weather caused a shift in behavior, that shift should be limited to certain regions). We seek out the opinions of informed sources while taking pains not to get sucked into conventional wisdom. We survey the situation on the ground and use that most important of statistical tools, common sense.

I'd like to say that the final step is testing the hypothesis, but that's not usually the way it works. When it comes to questions of causality, the final answer is usually just an educated guess. Fortunately, most of us have gotten to be pretty good guessers.

How does the Goldman Sachs hypothesis stand up to this (admittedly unscientific) approach?

For starters, the timing is all wrong.


(I'm assuming these are mostly likely-voter polls)

The Goldman Sachs-Whitman connection came out in February. The Abacus scandal hit big in mid-April. Poizner's Vulture ad was released at the end of April. Which of these would you associate with an inflection point in Whitman's support?

By comparison, the news for much of April had been dominated by Arizona's immigration law (which passed their house on April 13th) and since March, Poizner had been running ads attacking Whitman as being soft on immigration.



How about magnitude and precedent? Whitman took a fifty point lead down into the single digits. Immigration and RINOism are huge issues for the California GOP. Either could easily kill a campaign. The Goldman Sachs story, on the other hand, is abstract, relatively minor, and only tangentially related to the issues California conservatives care about.

And has anyone EVER burned off forty-plus points of lead because of this kind of business deal?

And finally, do any of the major players, the ones with access to internal polling, actually believe this hypothesis? Poizner clearly doesn't or he'd be making more than passing references to Goldman Sachs. Meg clearly doesn't or she wouldn't be spending her time insisting that she supports neither amnesty nor Senator Boxer. Hell, even the CDP doesn't or they wouldn't pick Peter Coyote to pitch their case. (I'm not saying that the CDP ads are ineffective; I'm saying they are targeted at the general election.)

To make this even less scientific, here's my good ol' boy take on what happened. It's the old story of an out-of-towner walking into a bar, hearing a couple of locals bragging and believing every word. ("You mean you really took down a $70 million dollar campaign?" "Yep, and we did it with just one little website.")

I suppose there's no harm in a little boasting (and it's not something that Democratic operatives get to do that often in California). I just hope that the people at TPM are a bit more worldly the next time the situation comes around.

And to close out the subject of nine pound hammers (and get our minds on something more pleasant than politics), here's a mental health break from a friend of mine:



Update: Ed Kilgore has a good analysis of the race here. I think he gives too much weight to the Goldman Sachs story but I may still just be stuck in argument mode as a reaction to TPM.

There's something about Olivia

Olivia Judson has a new piece on archaea, those strange life forms that can adapt to the harshest conditions on earth. I've read a number of other articles on the subject but most of them had an empty calorie quality, heavy on the gee-whiz and light on the substance. But Judson focuses on the good stuff. How do archaea differ from other life? What do we know or suspect about their evolution? Why have they been so difficult to study?

This is not an isolated case. I have often looked at the topic of one of Judson's pieces and thought "Not that again," but I always felt I owed her an apology by the time I got to the end.

Spousal Hires

I know that spousal hiring is a contentious topic and, like many complicated transactions, it is hard to carefully audit all elements of it. I am sure that situations happen that are not ideal. That being said, I do love it when I see a heart-warming story of it all working out and everyone (University included) being better off.

California election update -- the candidate's dilemma

Today, readers of Talking Points Memo (at least those in California) saw the following ad:


The election is (as previously mentioned here) a compound game requiring two consecutive wins. There are certain Pyrrhic strategies which maximize the chances of a win in the first part and minimize chances of a win in the second. (at the risk of putting too fine a point on it, immigration is the obvious Pyrrhic strategy in this game.)

Let's say W and P are players from the same party with an equal chance of winning the first part of the game. If they can agree not to go Pyrrhic, the winner of the first part has a good chance of winning the second. If one player goes Pyrrhic, then that player has a much better chance of winning the first part and a somewhat worse chance of winning the second, overall still a net improvement. If both go Pyrrhic, neither gains an advantage in the first part and both take a hit in the second.

Of course, in real life, we have all sorts of mechanisms like reputation and institutions such as party leaderships to help us avoid the prisoner's dilemma. That's not to say that it can't happen but we are able to discourage it.

Now consider this scenario. W is way ahead of P but is more vulnerable to the Pyrrhic strategy. Here P is almost certain to go Pyrrhic even though it will hurt his chances of winning the second part.

This suggests an interesting counter-intuitive strategy for W: open up a lead slowly and don't get too far ahead. (In other words, the opposite of the aforementioned blitzkrieg strategy.) There are external costs to going Pyrrhic, loss of reputation, long term damage to the party, pressure and disapproval from peers. W could hope to find that sweet spot where those costs slightly outweigh the benefits to P of a Pyrrhic/Pyrrhic contest.

I don't know if this would have worked for W (counter-intuitive strategies usually don't), but I have to think it would be better than what she came up with.

(note to the the picky: I'm used an exceptionally watered-down form of the word Pyrrhic here, but if you can't get sloppy in a blog, where can you get sloppy?)

Orphans in the Advocacy Economy

Monday night on the Big Bang Theory, Sheldon Cooper (Jim Parsons) lovingly cooed to his laptop that Ubuntu was his favorite Linux-based operating system. It was one of those pitch-perfect geek-chic allusions that the show specializes in, a reference to a piece of software beloved by computer nerds and all but unheard of among the surface dwellers.

I remember thinking that I had never heard anyone mention Ubuntu on television (I'm not even sure I've heard anyone talk about Linux), but I didn't give much thought to the underlying economics until the next day when I saw a piece from Forbes on what they termed 'rip-offs,' products and services with cheaper alternatives. It concluded with a section on basic cable:

The Rip-Off: All you want is basic cable, but your cable company wants you to have so much more--and pay through the nose for it. That's why it bundles in a whole mess of channels, including dozens that even the most feckless of couch potatoes won't watch.

How to Avoid It: Hulu.com offers thousands of videos, TV episodes (new and old) and full-length movies--all free. And Netflix charges as little as $9 a month for access to more than 100,000 TV episodes on DVD, as well as 12,000 movies.

I could sympathize with complaints about cable -- I had dropped it a few years earlier because they kept moving my favorite channels to more expensive tiers (TCM was the last straw) -- but I was surprised what the article didn't list.

I pay around $250 a year for pretty good high speed cable. Netflix starts at $60 a year and that limits you to two DVDs a month (I don't have an account). But the cheapest option, by far, is digital broadcasting (DTT). I bought a $45 dollar converter ($5 after coupon) and hooked it to rabbit ears I picked up at a 99 cents store. I now get over fifty channels with better quality than I used to get with cable, good enough to burn DVDs off the shows I record.

Ubuntu and DTT are orphans; they have no representation in markets where all their competitors have advocates. In theory, journalists are supposed to fill in the gap but that seldom works. As seen here, reporters generally limit their approach to repackaging the information and arguments generated by the advocates of the products they're writing about.

I use a lot of orphan products, mainly because I am very, very cheap and orphans tend to be great bargains. Ubuntu is free. DTT is free once you have a converter or a fairly new TV. Tap water is provided by my landlord. And when I was running a small business I generally found that no one could compete with the postal service on price.

Of course, these bargains indicate market inefficiency. This suggests an analogy to the legal system. We expect the courts to reach fair decisions given the condition that both parties have adequate representation. If we're going to expect market efficiency, perhaps we need to have a similar requirement.

Eggs

Nutritional epidemiology is a very complex area; eggs are a case in point. Evidence has been emerging that eggs are better for you than we had previously thought.

(h/t: Obesity panacea)

My SO (who does nutrition) has pointed out that we often look at just one part of a food and forget the other aspects. The result of simplistic recommendations may result in unbalanced diets -- a focus on fruits and vegetables can overlook important elements like adequate dairy intake. It's just hard to reduce to dietary recommendations to sound-bytes. Add in individual differences and the complexity of the problem increases (a lot).

Nutrition is a complex system with dozens of exposures (both macro- and micro-nutrients) that has significant issues with data collection. Food frequency surveys have known limitations and more sophisticated measures have their own limitations. Experiments are also difficult to design given the complexity of nutritional requirements across the lifespan.

It's not easy stuff!

Does Talking Points Memo know what it's talking about?

When it comes to California, apparently not. TPM (along with TNR) is one of my two favorite political sites but like many news organizations they tend to be NY/DC-centric and their quality tends to drop as they get further from home.

Here's the opening sentence from today's story:

Last week, California Republican gubernatorial candidate Meg Whitman released an ad that called Republican opponent Steve Poizner that dreaded "L" word -- liberal.

My first thought was that TPM was under the impression that Whitman just started calling Poizner a liberal; my second thought was that my first thought couldn't be right so I checked previous TPM stories on the race. Here's how they described the ad on May 11th:

Whitman Ad: My Conservative Opponent Is A Liberal!

California gubernatorial candidate Meg Whitman has a word she'd like you to associate with her GOP primary opponent: liberal.

In a new Whitman ad aimed at state Insurance Commissioner Steve Poizner, a narrator ominously asks, "How liberal is Steve Poizner?" before rattling off a long list -- He supported Al Gore! He supported higher sales taxes! -- before the narrator declares:

"Liberal on taxes. Liberal on spending. Just another liberal Sacramento politician."

As we reported earlier, a new poll shows Whitman's lead over Poizner in California's June 8 Republican gubernatorial primary at a very thin 39% to 37%.

How badly out of the loop was TPM here? For over a month, Meg Whitman has been running, in mind-numbingly heavy rotation, an attack ad with the tag line "Way more liberal than he says he is." As for that "long list," it wasn't very long, particularly considering that it is the exact same list that Whitman has been running (and running and running) in every single attack since February.

The TPM story suggests that these charges were a reaction to the race getting close despite the fact that they started when (according to TPM's own poll tracker) Whitman had a lead of over forty points and she pulling out further ahead.

It would have taken less than fifteen minutes of research to catch these problems. This is not the kind of reporting we expect from Talking Points Memo.

Candidate Rorschach will not be running

Nate Silver has an excellent post that implicitly makes a point that slips past most analysts: in a campaign where an incumbent faces an undetermined or little-known opponent, polls can be difficult to interpret. This doesn't mean that the polls at don't have useful information; it means you need to have someone competent (like Silver) to put them in the right context and properly analyze them.

CA election update -- the ad Meg Whitman didn't want to run

It's been almost a week since my last post on the subject and things have been moving along at a nice clip.

Goldman Sachs continues to show up in the discussion though mainly among Democrats. It has the makings of a potent issue in the general election but it's still a minor one in the primary.

The big issue for Republicans at the moment is the very one Whitman wanted to avoid: immigration. This is not a great issue for Poizner (he'd also like to hold onto the Latino vote), but it represents his best chance for a primary victory and he's hammering away at it and forcing Whitman into the never desirable "am not" mode of campaigning.

This weekend I was in a shop in Southern California where I heard the following ad over the radio:

Announcer: Meg Whitman on illegal immigration.

Meg Whitman: Don't be fooled by misleading ads, my position on immigration is crystal clear. Illegal immigrants are just that, illegal. I am 100 percent against amnesty for illegal immigrants. Period. As Governor, I will crack down on so-called sanctuary cities like San Francisco who thumb their nose at our laws. Illegal immigrants should not expect benefits from the state of California. No driver's license and no admission to state-funded institutions of higher education. And I'll create an economic fence to crack down on employers who break the law by using illegal labor.

Pete Wilson: This is former Governor Pete Wilson. I know how important it is to stop illegal immigration and I know Meg Whitman. Meg will be tough as nails on illegal immigration. She'll fight to secure our border and go after sanctuary cities. Please join me in supporting Meg Whitman for Governor.

Announcer: Paid for by Meg Whitman for Governor 2010. Meg2010.com.

The clientèle was largely Hispanic.

How to ace the essay section of the SAT

Write badly.





When I was teaching at a small urban prep school, we had a faculty meeting to discuss the writing section of the SAT. The people at College Board had provided a set of sample essays with grading guidelines. We individually scored each of the papers then got together to compare our results. There were some minor disagreements -- two of the essays were close in quality and we had trouble deciding which was best -- but there was one thing which we all agreed on: the one that the College Board listed as best came in a distant third.

The College Board's choice was terrible, but it was terrible in a distinct way that any English comp teacher would immediately recognize. The writer had tried to show his or her erudition by stuffing the essay with vocabulary words that weren't apt and literary allusions that didn't advance the argument. The prose was choppy, the sentences were clumsy, and the logic was flawed.

I remember discussing how we should handle this. Should we teach students how to write sharp, readable essays or should we tell them just to use the biggest words they could think of and shoehorn in a list of inappropriate literary references?

Now I know we should have added make it as long as possible.

(with thanks to Mr. Colbert for the tip.)

Financial Literacy

There is a really nice post over at Dean Dad about financial literacy. The part that I really liked was the issue of stock market returns. In the post he discusses how planning tools often assumed an 8% return while his own IRA has performed nominally negative. While it is possible that there could be a growth surge that makes an 8% return happen over a 30 year period, it has got to be one heck of a surge to cover up 12 years of negative nominal returns.

This is actually a more general principle and one that we should keep in mind as researchers -- for many phenomena that we study, there is limited data as to what could happen in the future. Extrapolating disease rates, for example, is hard based on current rates (especially for infectious diseases where exponential growth can matter). You also have subtle issues like vaccines and herd immunity (or, in a similar sense, behavioral firebreaks such as careful hand washing). Very small changes in circumstances can make it impossible to predict the future.

So my own lesson from this anecdote is that we need to be very skeptical about the assumptions that go into forecasting. We want to do it (because some information is better than no information) but we need to be ready to revise assumptions as data comes in.

How to ace the math section of the SAT

The SAT is the toughest ninth grade math test you'll ever take. The questions can be complex, subtle, even tricky in the sense that you have to pay attention to what the problem actually says, not what you expect it to say, but even the most challenging of the problems require no mathematical background beyond Algebra I and a few basic geometry concepts.

Because of this basic-math/difficult-questions dichotomy, many of those question have a short solution and a long one (check out this Colbert clip for an example). Quite a few others just have long solutions. Since the SAT gives students about ninety seconds for each problem, a high score normally indicates a student who was insightful enough to spot AHA! solutions* and fast enough to get through the rest.

Of course, if the SAT were not a timed test, a high score wouldn't indicate much of anything. Which is why the following is so troubling.

From ABC News:

At the elite Wayland High school outside Boston, the number of students receiving special accommodations is more than 12 percent, more than six times the estimated national average of high school students with learning disabilities.

Wayland guidance counselor Norma Greenberg said that it's not that difficult for wealthy, well-connected students to get the diagnoses they want.

"There are a lot of hired guns out there, there are a lot of psychologists who you can pay a lot of money to and get a murky diagnosis of subtle learning issues," Greenberg said. ...

The natural proportion of learning disabilities should be somewhere around 2 percent, the College Board said, but at some elite schools, up to 46 percent of students receive special accommodations to take the tests, including extra time.

This is not a new problem. I know from personal experience as a teacher that public schools have a history of trying to keep kids from being diagnosed as LD, both to save money and avoid paperwork. Everyone in education knew that, just as everyone knew that expensive private schools were working the system in the other direction.

* I assume everyone has read this. If you haven't, you should.

"Just when I thought I was out... they pull me back in."

I was going to to take a week off from blogging and work on another project, but this clip from the Colbert Report was too good to go unmentioned. Pay particular attention to the parts about learning disabilities and the grading of the writing section:

The Colbert Report	Mon - Thurs 11:30pm / 10:30c
Stephen's Sound Advice - How to Ace the SATs
www.colbertnation.com
Colbert Report Full Episodes Political Humor Fox News

"Red and yellow can kill a fellow"

The pharmaceutical industry does a great job developing most kinds of drugs, but there are certain areas where we might consider a less market based approach.* Antibiotics are one example. Here's another:

Unfortunately, after Oct. 31 of this year, there may be no commercially available [coral snake] antivenom (antivenin) left. That's the expiration date on existing vials of Micrurus fulvius, the only antivenom approved by the Food and Drug Administration for coral snake bites. Produced by Wyeth, now owned by Pfizer, the antivenom was approved for sale in 1967, in a time of less stringent regulation.

Wyeth kept up production of coral snake antivenom for almost 40 years. But given the rarity of coral snake bites, it was hardly a profit center, and the company shut down the factory that made the antivenom in 2003. Wyeth worked with the FDA to produce a five-year supply of the medicine to provide a stopgap while other options were pursued. After that period, the FDA extended the expiration date on existing stock from 2008 to 2009, and then again from 2009 to 2010. But as of press time, no new manufacturer has stepped forward.

Antivenom shortages are a surprisingly common occurrence. The entire state of Arizona ran out of antivenom for scorpion stings after Marilyn Bloom, an envenomation specialist at Arizona State University, retired in 1999. Bloom had been single-handedly making all the scorpion antivenom for state hospitals. Recently, Merck & Co, the only FDA-licensed producer of black widow antivenom, has cut back distribution because of a production shortage of the drug. In a 2007 report, the World Health Organization listed worldwide envenomations as a "neglected public health issue."

New scorpion and black widow antivenoms are currently in the pipeline, thanks to efforts by several poison-control associations to speed foreign drugs into the market through FDA research programs. There is also a coral snake antivenom produced by Mexican drug manufacturer Instituto Bioclon that researchers believe could be even more effective and safe than the outgoing Wyeth product. But that drug, Coralmyn, is not currently licensed for sale by the FDA. The tests required for licensing would cost millions of dollars, and for such a rare treatment (there are 15 times as many scorpion stings per year as coral snake bites), it could take decades for Bioclon to make its money back.

*Or perhaps, to go the other way, a less regulated one. I'm open to suggestions.

Another failure for the blitzkrieg strategy? -- updated

This is getting interesting. There's a new poll out in California that shows Meg Whitman leading Steve Poizner 39 to 37. Now it's always nice to be ahead (even if it is within the margin of error) but there are two small details that might undercut Whitman's sense of triumph:

1. She spent $70,000,000 establishing that two point difference;

2. It was a forty-nine point lead when she started.

This has been a fun campaign to follow (who among us is above a little schadenfreude at the sight of an arrogant billionaire getting blindsided?), but it has also been interesting from a strategic point of view, raising all sorts of neat questions about the best way of runninga campaign for a compound election, particularly if you happen to be a Republican in a blue state in 2010.

Strategies in simple elections are, well, pretty simple: you pick the course of action that gets you the most votes. There are actually two goals here -- you want to maximize both the size of your mandate and your chances of winning -- but they generally line up so well that we can treat them as one.

Strategies for compound elections are anything but simple. For starters you can no longer count on chances of winning and size of mandate lining up. Mandate becomes a secondary goal that has to be somehow weighed in. Worse yet, the probability of winning is now the product of the probabilities of winning two different elections, each of which has a different (in some cases radically different) optimal strategy.

Richard Nixon solved this problem with a hard pivot strategy -- run as far to the right as you can during the primary, then run as fast as you can to the center in the general election. He based this on the observation that Republicans tended to be much more loyal to their candidates. Once you had the primary under your belt, you could generally count on their votes. (Nixon had also seen what happened when Goldwater had gone hard right and stayed there in '64.)

Nixon's insight was brilliant. It was not, however, all that robust. Republican core voters will no longer tolerate movement to the center, even when their candidate is trailing badly. This leaves GOP candidates in blue and purple states with a serious problem. The Nixon strategy is forbidden; the Goldwater strategy is hopeless.

For a while, it looked like a blitzkrieg strategy might work. This approach involves coming in with a big war chest and a stack of endorsements, establishing a huge lead as early as possible, marginalizing primary competitors, and running a campaign for a de facto simple election. Charlie Crist and Meg Whitman both tried this approach and though it is possible that either or both might end up winning in the general election, it is safe to say that the blitzkrieg failed.

Crist has adopted what you might call a Gordian-knot strategy and skipped directly to the simple election. I don't see that option for Whitman. She can continue carpet-bombing the state with ads while not taking a position on anything that doesn't involve tax-cuts. No polls to date show Poizner ahead. It's possible she could scrape by without a course change.

Or she could try a hard right turn and match Poizner's positions on gay marriage and the Arizona immigration law, but it's awfully late in the game to try to win over social conservatives and these stances will be hard to live down in a general election. Still, if the trend lines continue to hold, she may not have a choice.

Update: Talking Points Memo has a completely different take on Poizner's surge, crediting it mostly to Whitman's association with Goldman Sachs. I suspect that the difference has a lot to do with geography. TPM is based in New York where it's easy to imagine the world revolving around the financial services industry. I'm sitting in Whittier, where I've noticed the following:

1. Various groups have been trying to push the Goldman Sachs story for months;

2. By contrast, the Poizner campaign has only gone into high gear recently (just before Whitman started to slide);

3. Goldman Sachs continues to play a small role in the campaign. The "Vultures" ad (which debuted April 30) is not in particularly heavy rotation. It's hard to see how it could have slashed Whitman's lead in about a week;

4. People seem to be paying more attention to more prominent ads featuring immigration, McClintock's endorsement, and Arnold Schwarzenegger morphing into Meg Whitman.

TPM is still my favourite place for politics, but I think they got this one wrong.

"U-Va. should fight Cuccinelli's faulty investigation of Michael Mann"

The Washington Post weighs in on Ken Cuccinelli's investigation (via Mark Thoma):

By equating controversial results with legal fraud, Mr. Cuccinelli demonstrates a dangerous disregard for scientific method and academic freedom. The remedy for unsatisfactory data or analysis is public criticism from peers and more data, not a politically tinged witch hunt or, worse, a civil penalty. Scientists and other academics inevitably will get things wrong, and they will use public funds in the process, because failure is as important to producing good scholarship as success. For the commonwealth to persecute scientists because one official or another dislikes their findings is the fastest way to cripple not only its stellar flagship university, but also its entire public higher education system.

I'm not saying we should stop trying...

but it is worth remembering that most innovations don't work out all that well.


From Wired:

What Felix Salmon missed -- the Hulk paradox

Felix Salmon has an interesting piece in the New York Times arguing that:

[A] futures contract on box office receipts would be great news for the industry. For one thing, if the market got big enough, it would allow studios to easily hedge their investments in movies just by entering into a simple derivatives transaction. Studios could essentially sell contracts on their movies’ grosses into the open market, and pocket the proceeds. They would lose money on the contract if the movie does well, but in that case they’d make enough money on the movie itself to cover their derivatives losses.

The problem Salmon's thesis is that Hollywood runs on expectations and perceptions. A futures market would add another set of hard-to-dismiss numbers to the process. It's difficult to spin a forty million dollar opening as success if the market had predicted seventy.

Consider the case of the Hulk movies. Most people (myself included) were under the opinion that the first was a flop and the second was a success, but the first film made almost exactly the same amount of money when you take into account its smaller budget. In terms of return on investment it did much better than the second film, but in terms of advancing careers, it did much worse.

This was largely due to the fact that the press had reported an expected opening of $45 million (weak for a $150 million action film). By comparison, the $55 million opening looked great even though the first had opened with $62 million five years earlier with a smaller budget. The ability to get that '45' into the press made the film a perceived success. If there had been a futures market and it had said '55,' the film would have been seen as lackluster at best.

I'll let the Onion take it from here:

Why No One Want Make Hulk 2?
By The Hulk

July 14, 2004 | ISSUE 40•28

X2 come out last year. Spider-Man 2 come out last month. Both great sequels to great movies about Hulk friends. Hulk love great action movies about friends! People buy tickets. Make money for theaters, make money for movie company. Movie company make more movies with money. Already, they working on X-Men 3. Hulk movie come out last year. It success. It big popcorn movie with heart. So why no one want make Hulk 2? It make Hulk mad!

Hulk know what people say. Original movie no good, people say. Hulk movie Hulk-sized bomb, people say. That not true! Hulk more successful than people think. Make $132 million in U.S. alone, only cost $120 million. That not small potatoes. Add international box-office receipts and DVD sales and it add up to big money. Big! Oh, and did Hulk forget merchandising tie-ins? First Hulk movie really forge Hulk brand identity. Make people aware of Hulk. Hulk now poised to build on success of first Hulk movie. Hulk 2 smash box-office records!

(you really have to read the rest of this)

White flight

The most dramatic improvement I've ever seen in a marketing model came from adding an urban/suburban/rural variable. I wonder how that model is holding up given this:

In a reversal, America's suburbs are now more likely to be home to minorities, the poor and a rapidly growing older population as many younger, educated whites move to cities for jobs and shorter commutes.

An analysis of 2000-2008 census data by the Brookings Institution highlights the demographic "tipping points" seen in the past decade and the looming problems in the 100 largest metropolitan areas, which represent two-thirds of the U.S. population.

The findings could offer an important road map as political parties, including the tea party movement, seek to win support in suburban battlegrounds in the fall elections and beyond. In 2008, Barack Obama carried a substantial share of the suburbs, partly with the help of minorities and immigrants.

"You can't shrink your way into prosperity"

This is interesting:

New research from the Univ. of Colorado, Denver shows major job cuts don't guarantee prosperity down the line but, in fact, lead to lower profits and stock returns.

"Those who cut deepest, relative to industry peers, delivered smaller profits and weaker stock returns for as long as nine years after a recession," the WSJ reports, citing a study by Univ. of Colorado business professor Wayne Cascio. He studied how companies in the Standard & Poor's 500-stock index have performed during and after recessions over an 18 year period.

Why?

Turns out firms that cut aggressively aren't prepared to ramp up quickly once the recovery begins. In contrast, peer firms that cautiously trimmed are well-staffed to take advantage of a swift shift in momentum. "You can't shrink your way into prosperity," says Cascio in the article.

Assuming the research holds up, isn't this the exact sort of thing an efficient market should catch?

A brief history of beast-starving by Bruce Bartlett

Via Thoma, Bruce Bartlett traces the rise and consequences of an economic theory. The whole thing is worth a read but these paragraphs in particular caught my eye:

Once upon a time Republicans thought that budget deficits were bad, that it was immoral to live for the present and pass the debt onto our children. Until the 1970s they were consistent in opposing both expansions of spending and tax cuts that were not financed with tax increases or spending cuts. Republicans also thought that deficits had a cost over and above the spending that they financed and that it was possible for this cost to be so high that tax increases were justified if spending could not be cut.

Dwight Eisenhower kept in place the high Korean War tax rates throughout his presidency, which is partly why the national debt fell from 74.3% of gross domestic product to 56% on his watch. Most Republicans in the House of Representatives voted against the Kennedy tax cut in 1963. Richard Nixon supported extension of the Vietnam War surtax instituted by Lyndon Johnson, even though he campaigned against it. And Gerald Ford opposed a permanent tax cut in 1974 because he feared its long-term impact on the deficit.

Tyler Cowen has nerves of steel

With no apparent sign of fear or hesitation, he links to the following WSJ article:

Do Girls Speed More Than Boys?
Survey Says Girls Drive More Aggressively; Insurers Up Rates
By JOSEPH B. WHITE and ANJALI ATHAVALEY

Some big auto insurers are raising the rates they charge to cover teenage girls, reflecting the crumbling of conventional wisdom that young women are more responsible behind the wheel.

In a survey of teenage drivers, Allstate Insurance Co. found that 48% of girls said they are likely to drive 10 miles per hour over the speed limit. By comparison, 36% of the boys admitted to speeding. Of the girls, 16% characterized their own driving as aggressive, up from 9% in 2005. And just over half of the girls said they are likely to drive while talking on a phone or texting, compared to 38% of the boys..

The results were "a surprise to many people," says Meghann Dowd of the Allstate Foundation, an independent charitable organization funded by Allstate which sponsored the survey.

While teens fessed up about their own bad behavior, they also said their friends drive even worse. The study found that 65% of the respondents, male and female, said they are confident in their own driving skills, but 77% said they had felt unsafe when another teen was driving. Only 23% of teens agree that most teens are good drivers. This suggests teens recognize in their friends the dubious and dangerous behavior they won't admit to indulging in themselves.

The data was gleaned from online interviews with 1,063 teens across the country. It was conducted in May 2009 for the Allstate Foundation by the TRU division of TNS Custom Research Inc., a Chicago-based youth research and marketing firm. (For highlights of the study, see www.allstate.com/foundation/teen-driving/Shifting-Teen-Attitudes.aspx).

The survey relies on what teens report about themselves, and Allstate Foundation spokeswoman Meghann Dowd says that means the results could be affected by how forthcoming individuals are when answering the survey questions.

(read the rest here)

I don't know how he does it. I started to sweat when I hit the second paragraph and it just got worse from there. I'm not ashamed to admit that I get nervous around studies that:

1. assume that teenage boys and teenage girls are equally likely to tell the truth about their illegal driving habits;

2. assume that teenage boys and teenage girls are equally likely to be aware of their illegal driving habits;

3. have a spokesperson who talks about individuals not being forthcoming but doesn't mention the possibility that this effect might be more prevalent for one gender;

4. don't cross-check their findings against highly correlated hard data like highway patrol citation records;

5. contain at any point any variation of the phrase 'online interview.'

I have to sit down now. I'm getting dizzy...

Things I've learned -- 60s pulp edition

One of the pleasures of popular art is the glimpse it provides into the attitudes of the times. Sometimes this pleasure comes from the naivete of earlier times. More often, though, the experience is just the opposite -- you discover that the attitudes of previous generations were complex, surprisingly modern and often at odds with the conventional view of the era.

I recently read a couple of pulp novels from the Sixties that fell into the second category. One was The Thief Who Couldn't Sleep by Lawrence Block (discussed here). The other was The Ambushers, the sixth book in the Matt Helm series by Donald Hamilton.

Matt Helm was a counter espionage agent specializing in wet work, closer to a John LeCarre hero than to James Bond. The books were well reviewed (Anthony Boucher, writing, I believe, in the New York Times, said "Donald Hamilton has brought to the spy novel the authentic hard realism of Dashiell Hammett; and his stories are as compelling, and probably as close to the sordid truth of espionage, as any now being told."), they were extremely popular and they were sturdy enough to survive a god-awful series of in-name-only adaptations starring Dean Martin.

If you picked up a copy of the Ambushers in 1963 and turned to the first page, you'd find the narrator slipping into a Latin American country carrying a rifle with a high powered scope. The rebel leader he has been sent to kill may have had it coming, but it's not obvious that target was any worse than the leader the U.S. supported (later in the story, Helm is glad to hear that the latter has also been assassinated though his superiors most certainly are not).

The Russians in the Ambushers are sometimes enemies and sometimes allies, depending on the circumstances. The American agents generally hold the high moral ground but it's a distinction that can be rather thin and the protagonist has learned not to dwell on it too much. Everyone's hands are dirty.

As in many stories of the period, the threat nuclear war here comes not from either of the superpowers but from a third party. In this case, a Nazi war criminal who, not surprisingly, would like to see Russia and America destroy each other.

You will often hear the attitudes implicit in this story associated with the late Sixties and early Seventies, usually attributed to the escalation of the war and and the rise of a politicized youth culture, but this was 1963. It was early in the war and even the oldest of the boomers were still in high school.

If you surveyed the pop culture of the time you would find other evidence that a radical shift in the way we looked at the cold war took place in the late Fifties and early Sixties. Nuclear war was no longer likely to be depicted as a Pearl Harbor-style attack but rather as either a horrible mistake (Failsafe -- novel 1962, film 1964) or the work of a madman (Red Alert -- 1958/Strangelove* -- 1963) or the terrorist scheme of a third party (too numerous to mention). Anti-communist agents could be as morally compromised as the enemy (The Spy Who Came in from the Cold -- 1963). A sympathetic Russian was so acceptable that by 1964, you could have a loyal and openly communist Russian agent as a co-star in a popular spy show (albeit a Russian played by the Scottish David McCallum).

I've wondered if this shift was a reaction the Cuban Missile Crisis. Talking to those who were around at the time, I get the impression that for all the paranoia and anxiety and civil defense drills, the concept of nuclear war was never truly real to most people until 1962.

Good researchers in sociology or political could probably provide a rigorous answer to the question of what exactly drove the shift. I'd be interested in seeing what they came up with and if they need another topic after that, I have a whole shelf Gold Medal paperbacks for them to check out.

(You can watch Dr. Strangelove online here)

Things I've learned -- media critic edition

Felix Salmon has a good, thoughtful take on the Newsweek sale, but before I link to it, I'd like to preface it with a few things I've learned writing about media for the past twenty years. They aren't specifically at Salmon's piece but they are relevant to the larger discussion:

1. Profits from newspapers and broadcasting were obscenely high for so long that anything less now seems like the end of the world;

2. Markets for traditional media are shrinking but the rate of decline is generally overstated by shoddy statistics;

2. Most failures in media are blamed on changes in markets and technology;

3. Most failures in media are caused by unrealistic business plans, poor management, and products that suck;

4. When reading expert opinions on media and business, remember that many of these experts predicted the death of networks in the Eighties and the success of online grocery shopping in the Nineties; these experts have lost none of their self-confidence.

Felix Salmon, on the other hand, is one of those experts whose confidence level is highly correlated to his track record. Check out his very shrewd analysis here.

Scraping by on 200 million

The LA Times recently ran an interesting article on attempts to reign in the costs of big budget movies:

Bruckheimer's next production, "Pirates of the Caribbean: On Stranger Tides" — his 26th film for Disney in 16 years — dramatically illustrates the new reality.

With the fourth installment of the swashbuckling tale poised to start shooting June 14, Bruckheimer and the filmmakers are scrambling to meet the more constrained budget that Disney is imposing. Although it's still large — north of $200 million — it is at least a third less than the last "Pirates" movie and includes far fewer shooting days and visual effects shots.

Ross said that Bruckheimer, known for spending lavishly, is working within the new constraints. Bruckheimer reassured him of that in Ross' first week as studio chief, Ross said: "He looked at me and said, 'I will work with you to figure out the economics of the movies going forward because I understand what we are all facing.' And I said two words: thank you."

Even before Ross took the studio reins last fall, Walt Disney Co. Chief Executive Bob Iger was mandating that executives ratchet down costs after underperforming movies — including Bruckheimer's expensive family film "G-Force" — triggered two quarterly losses at the studio last year.

It's a well-reported piece but it doesn't address the underlying question of why movie budgets have skyrocketed. Even if you adjust for inflation, 23 of the 25 most expensive films ever made were produced in the past fifteen years, and yet the same period was marked by economic and technical trends that should have made production less, not more expensive.


CGI and digital production

CGI has become so associated with big budgets that it's easy to forget that it is primarily a cost-saving measure. Ray Harryhausen used to tell directors that he could give them anything they wanted if they had the time and the money. That might not have been completely true but for the vast majority of the cases, CGI is used to replace the more expensive techniques of effects men like Harryhausen or to avoid costly reshoots (as in Disney's digital breast reduction of Lindsay Lohan in Herbie). Add to that the cost and time savings of innovations like nonlinear editing and you would expect film-making to get much cheaper.


Weakened unions

Hollywood is still a heavily unionized place but these unions have been steadily losing ground for at least the past quarter century. You would expect some of those losses to translate into lower production costs.


Competition from other states and countries

There have been many attempts to grab some of the business away from Southern California. Most of those attempts were sweetened with so many tax breaks and other incentives that these would-be competitors often lost money on the deals.


Off-Shoring

As a consequence of digital production, a great deal of work can now be done remotely.


Low inflation

Prices in general haven't exactly been shooting up over the past twenty years.


Obviously something caused the surge, but I suspect it has less to do with economic forces and more to do with studio politics.