This should literally be a textbook example of how phrasing and context affect polling

Talking Points Memo discusses how two polls given in the same week can get starkly different results:

Last week, a Gallup poll showed that 60% of Americans would rather see a budget compromise than see members of Congress who represent their interests hold out for their ideal budget, if it means the government would shut down. That phrases the current debate in Washington fairly concisely.

Compare that to Rasmussen, which framed the question much differently:

5* Would you rather have Congress avoid a government shutdown by authorizing spending at the same levels as last year or would you rather have a partial government shutdown until Democrats and Republicans can agree on what spending to cut?

That frames the budget showdown as an either or: either the government continues spending at current levels, or it shuts down until cuts are made. In response to that, 58% of likely voters said they preferred a government shutdown.

Rasmussen is, of course, something of a special case. To understand just how special, take a look at Nate Silver's excellent analysis,"When ‘House Effects’ Become ‘Bias.’"

"The Economics of Blogging and The Huffington Post"

After the election season, my regular visits to FiveThirtyEight tapered off then simply came to a stop.

That might have been a mistake on my part.

(thanks again to Felix Salmon)

Bad polls, good polls, better polls

You might have expected Nate Silver to sleep through the rest of the week after the elections, but the Red Bull must flow freely at 538. Anyone who is serious about polls should read these two recent posts:

Rasmussen Polls Were Biased and Inaccurate; Quinnipiac, SurveyUSA Performed Strongly

and

When ‘House Effects’ Become ‘Bias’

Even more interesting is Mark Blumenthal's article on the remarkable accuracy of both camps internal polling in Nevada.

Not All Polls Were Wrong In Nevada

Nervous Nate -- why timidity at FiveThirtyEight is a sign of professionalism

I've been working on an article on educational statistics. One of the major themes is the way statisticians should handle research when they don't have much faith in the results. This is not quite the same as the way Bayesians assign weights to beliefs but it's not all that different. Us poor, simple frequentist-folk approach the problem in an informal way and try to present our results with sufficient caveats that the listeners will understand that we may have a small p-value but that doesn't mean we'd bet the farm.

As I was working on that part of the article, I was struck by how Nate Silver was doing exactly what I was describing, pointing out the factors that could throw his forecasts off, reminding people that in this election, more than most, anything could happen.

That's one of the reason's Silver is the gold standard* in the field.





* Sorry, I just couldn't help myself.

Nate Silver tries to disabuse political reporters of another favorite myth

In today's 538, Nate Silver takes down the myth that polls tend to show momentum. He left out some details that OE readers might be curious about (but, of course, we're not exactly the target audience), but it's an excellent piece of statistical writing.

The Misunderstanding of Momentum

By NATE SILVER

Turn on the news or read through much of the analysis put out by some of our friends, and you’re likely to hear a lot of talk about “momentum”: the term is used about 60 times per day by major media outlets in conjunction with articles about polling.

When people say a particular candidate has momentum, what they are implying is that present trends are likely to perpetuate themselves into the future. Say, for instance, that a candidate trailed by 10 points in a poll three weeks ago — and now a new poll comes out showing the candidate down by just 5 points. It will frequently be said that this candidate “has the momentum”, “is gaining ground,” “is closing his deficit,” or something similar.

Each of these phrases are in the present tense. They create the impression that — if the candidate has gone from being 10 points down to 5 points down, then by next week, he’ll have closed his deficit further: perhaps he’ll even be ahead!

There’s just one problem with this. It has no particular tendency toward being true.

Read the rest...

Candidate Rorschach will not be running

Nate Silver has an excellent post that implicitly makes a point that slips past most analysts: in a campaign where an incumbent faces an undetermined or little-known opponent, polls can be difficult to interpret. This doesn't mean that the polls at don't have useful information; it means you need to have someone competent (like Silver) to put them in the right context and properly analyze them.

Nate SIlver debunks another polling myth

Here's the old chestnut (from Robert Moran):

In a two way race, political professionals don't even bother to look at the spread between the incumbent and the challenger, they only focus on the incumbent's support relative to 50%. Incumbents tend to get trace elements of the undecideds at the end of a campaign. Sure, there is the occasional exception, but this rule is fairly ironclad in my experience.

Here's Silver's takedown:

There are several noteworthy features of this graph:

1) It is quite common for an incumbent to be polling at under 50 percent in the early polling average; this was true, in fact, of almost half of the races (30 of the 63). An outright majority of incumbents, meanwhile, had at least one early poll in which they were at under 50 percent of the vote.

2) There are lots of races in the top left-hand quadrant of the graph: these are cases in which the incumbent polled at under 50 percent in the early polling average, but wound up with more than 50 percent of the vote in November. In fact, of the 30 races in which the incumbent had less than 50 percent of the vote in the early polls, he wound up with more than 50 percent of the vote 18 times -- a clear majority. In addition, there was one case in which an incumbent polling at under 50 percent wound up with less than 50 percent of the November vote, but won anyway after a small third-party vote was factored in. Overall, 19 of the 30 incumbents to have less than 50 percent of the vote in the early polling average in fact won their election.

3) 5 of the 15 incumbents to have under 45 percent of the vote in early polls also won their elections. These were Bob Menendez (38.9 percent), Tim Palwenty (42.0 percent), Don Carcieri (42.3 percent), Jennifer Granholm (43.4 percent) and Arnold Schwarzenegger (44.3 percent), all in 2006.3b) If we instead look at those cases within three points of Ted Strickland's 44 percent, when the incumbent had between 41 and 47 percent of the vote in early polls, he won on 11 of 17 occasions (65 percent of the time).

4) Almost all of the data points are above the red diagonal line, meaning that the incumbent finished with a larger share of the vote than he had in early polls. This was true on 58 of 63 occasions.

4b) On average, the incumbent added 6.4 percent to his voting total between the early polling average and the election, whereas the challenger added 4.5 percent. Looked at differently, the incumbent actually picked up the majority -- 59 percent -- of the undecided vote vis-a-vis early polls.

4c) The above trend seems quite linear; regardless of the incumbent's initial standing in the early polls, he picked up an average of 6-7 points by the election, although with a significant amount of variance.

5) The following corollary of Moran's hypothesis is almost always true: if an incumbent has 50 percent or more of the vote in early polls, he will win re-election. This was true on 32 of 33 occasions; the lone exception was George Allen in Virginia, who had 51.5 percent of the vote in early polls in 2006 but lost re-election by less than a full point (after running a terrible campaign). It appears that once a voter is willing to express a preference for an incumbent candidate to a pollster, they rarely (although not never) change their minds and vote for the challenger instead.