The very idea of claiming a candidate a year before an election has X% chance of winning is gross statistical malpractice.

[This has been sitting in the queue for a while but I think it still has another month or two on the sale-by date.]

A couple of issues make talking about predictive modeling difficult: 

Predictive range -- When we say someone accurately predicted an outcome, are we talking about an event that happened the the next day or the next year? Most are easier in the short range. Some are easier in the long (we'll all be dead) range. This has been particularly relevant with poll-based electoral predictions, where the track record for short term models has been great and long term models has been disastrous. We have an extensive history of pundits bragging about successes in the first category while hoping you'll forget about their failures in the second.

So, Is Obama Toast? by Nate Silver

Then there's modelers' luck. The problem with checking any probabilistic claim is that being right (got the outcome predicted) doesn't mean you were right (used a sound approach to estimate reasonable odds). The person who told you not to try to fill an inside straight was right and the person who told you to go for it was wrong, even if you did end up getting the card you were looking for.  

Back in 2011, Nate Silver said that, unless there was a major uptick in the economy, Obama had very little chance (think Russian Roulette odds) of winning the election. Instead, the economy was basically flat and yet the incumbent not only won but won by a comfortable margin. It is safe to say the model is wrong but was it bad or merely unlucky? Based on this article's long and admirably transparent explanation, I have to go with bad and here are some of the reasons why.

The fundamental assumption of predictive modeling is that things still work like they used to. Correlations and causal relationships from the past still hold. Data are collected in roughly the same way and the statistics derived from them have the same definitions.

The first practical implication of the fundamental assumption is that you can't push the boundaries of your data back too far. If things were two different beyond a certain point, you can't reasonably assume that they will generalize to today.

How far back you can reasonably go depends on what kinds of questions you are trying to answer and what types of data you're relying on. In terms of re-elections, 1931 is certainly too far back for any kind of meaningful comparison. This would have been 80 years before Nate Silver did his analysis which is a long time with respect to making political or social comparisons. More importantly, the way public opinion was formed and measured is enormously different. Add to that the huge outlier which was the beginning of the Great Depression.

We are even further into outlier territory with the entire presidency of FDR, especially if we're talking about the concept of re-election. (Silver goes back to 1944 in his analysis.) Truman is also problematic for a number of reasons, not the least of which being the fact he was not technically re-elected. The same concerns apply to LBJ and Gerald Ford.

This leaves us with Eisenhower, Nixon, Carter, Reagan, HW Bush, Clinton, and George W Bush. 

N equals 7.

Even if we ignore the distinction between election and reelection (which is a pretty big jump) and look at all elections going back to 1952, which is about the maximum I would be comfortable with, we're still looking at 15 elections to take us to Obama versus Romney. 

N equals 15.

(If we were just looking at win/loss, one of those 15 data points is missing since we will never know who actually won the 2000 election.)

That would be a small sample under the best of circumstances, but in this case we also have messy data, major one time events like the Cuban Missile Crisis, the Vietnam War, the Watts riots and the Iranian hostage crisis, not to mention waaaaaaay more than 15 researcher degrees of freedom.

Case in point. Look at how Silver handles the 800 lb gorilla of the model.

A president’s approval rating at the beginning of his third year in office has historically had very little correlation to his eventual fate. In January 1983, Reagan had an approval rating of just 37 percent, but he won in a landslide. George H. W. Bush had a 79 percent approval rating in January 1991 and was soundly defeated. But voters start to think differently about a president over the course of his third year; they view him more on the basis of his performance and less on the hopes they had for him. These perceptions are sharpened by the beginning of the opposition party’s primary campaign, which, of course, accentuates the negatives.

A president’s approval rating toward the end of his third year, therefore, has been a decent (although imperfect [I love how Silver throws in these little qualifiers while getting further and further ahead of the data -- MP]) predictor of his chances of victory. Reagan saw his approval rating shoot up to 51 percent in November 1983 amid the V-shaped recovery from the recession of the previous year — the first sign that he was headed for a big win. Obama’s approval rating may have rebounded by a point or two from its lows after the debt-ceiling debacle — but not by much more than that. In late October, it ranged between 40 and 46 percent in different polls and averaged about 43 percent.

Look at the forks. Of the various factors we can put in the model,  we pick approval rating but the fit to our fourteen data points is still crappy, so we limit ourselves to an arbitrary interval. Silver tells a good story to justify setting the the cut-off at the end of the third year, but that's all it is, a story, and even if it's true, we have no way of knowing if that particular cut-off will be appropriate going forward.

Silver also considered

The good news is that voters have short memories. If there are hopeful signs during an election year, they may be willing to forget earlier problems. Reagan, Nixon, Eisenhower and Truman all won despite recessions earlier in their terms. Moreover, voters’ evaluations of the economy are relatively forward-looking. Even if the economy is below its full productive capacity — as it was in November 1984 when the unemployment rate was 7.2 percent, and as it certainly was in 1936, when it was still around 17 percent — voters may be willing to overlook this, provided it seems headed in the right direction.

[Though it's a bit off topic for this post. It is worth noting that Nate Silver who has become one of the leading voices in the why aren't the Democrats panicking contingent has completely reversed the position stated here. If voters really do have short economic memories and are inclined to heavily weight positive trends, Joe Biden should be sitting awfully pretty now according to the Nate Silver of 2011.]

The bad news for Obama is that he has already missed his opportunity for a V-shaped recovery, and the prospects for a U-shaped recovery seem uncertain. In October, a panel of economists polled by The Wall Street Journal forecast 2.3 percent G.D.P. growth (adjusted for inflation) in 2012, somewhat below the election-year average of 3 or 4 percent and only enough to provide for modest job creation.

To his credit, Silver did explicitly state most of his assumptions and even assigned different probabilities to different scenarios.

CASE STUDY NO. 1: ROMNEY AND STAGNANT ECONOMY
Obama approval rating in November 2011: 43% 
G.D.P. growth in 2012: 0%
Probability of winning the popular vote: Romney: 83%, Obama: 17%

We begin with the worst of these situations for Obama: Mitt Romney is the Republican nominee, and economic growth, rather than continuing along sluggishly, comes to a halt (perhaps the debt dominoes have fallen in Europe). Under these assumptions, Obama would only have a 17 percent chance — about one in six — of winning a majority of the popular vote.

His chances are slim enough in this case that if I woke up next November to discover that we would have four more years of Obama, I might ask whether there was some sort of October surprise: “Mitt in Torrid Affair With Filipina Housekeeper.” Subhead: “Illegal Immigrant Got Free Romneycare.” Then I might ask if Sarah Palin had run on the Tea Party ballot line and taken 6 percent of the vote.

...

 

In practice, voters may think about the economy as falling into one of three basic categories — Good, Bad and Getting Better — rather than along a continuum. Obama would benefit if he could make a credible case for Getting Better, something he would not be able to do in this situation. But since he’s already unable to make that case now — remember “Recovery Summer”? — it’s plausible that a deterioration in the numbers would not hurt him as much as an acceleration of growth might help him. Beating Romney with 0 percent growth would not be easy, but it might not be that much more difficult than beating him with 2 percent growth (also no piece of cake, of course).

CASE STUDY NO. 2: ROMNEY AND IMPROVING ECONOMY
Obama approval rating in November 2011: 43%
G.D.P. growth in 2012: 4%
Probability of winning the popular vote: Romney: 40%, Obama: 60%

Obama would be far better off if he could make the Getting Better case. Imagine, as before, that Romney is the nominee. But rather than going into recession, the economy grows by 4 percent next year, enough to make a real dent in the unemployment rate. This would be enough to make Obama the favorite.

But not by all that much: he’d have only about a 60/40 edge.

Why not larger? The key to understanding this one is that Obama has a lot of gravity to overcome. Voters usually put their earlier concerns aside if there is an improvement in the economic fundamentals in the election year. But there have been exceptions: growth was quite strong in 1992, but voters were still punishing George H. W. Bush for the 1990-91 recession and the jobless recovery it produced.

...

Still, the most likely eventuality in this case — enough economic growth that the White House gets to make the Getting Better case while maintaining a straight face — is a narrow win for Obama. Perhaps it would be somewhat like the one that George W. Bush secured in 2004: it would keep the network anchors up late, but it wouldn’t be close enough to put us in Recount Land.

This breakdown highlights on of the most curious aspects of Silver's model. He does not appear to have run numbers based on current economic numbers or trends (despite citing that "bad news" WSJ forecast turned out to be pretty much dead on the money). That 17% was based on the assumption of the economy getting much worse in 2012. The narrow win scenario was based on it getting much better. Neither happened...

... but Obama not only won without anywhere near the 4% GDP growth of the second scenario, he did it with a 3.9% margin and managed to get the network anchors to bed at a reasonable hour.

Silver's model could have been better but they weren't terrible -- we've seen worse -- but it never should have been built in the first place. Predictions about a presidential election serve no public good. Long term predictions about a presidential election have the added charm of being about as reliable as a magic 8-ball.