Four years ago (more or less) -- something to remember when reading the latest analysis from Nate Cohn (and a lot of others)

Track records matter. Like it or not, unless you're actually working with the numbers, you have to rely to some degree on the credibility of the analysts you're reading. Three of the best ways to build credibility are:

1. Be right a lot.

2. When you're wrong, admit it and seriously examine where you went off track, then...

3. Correct those mistakes.

I've been hard on Nate Silver in the past, but after a bad start in 2016, he did a lot to earn our trust. By the time we got to the general election, I'd pretty much give him straight A's. By comparison, there were plenty of analysts who got straight F's, and a lot of them are playing a prominent role in the discussion this time around..

Tuesday, September 29, 2015

Revisiting Nate Cohn -- Scott Walker edition

I was going back and forth on whether or not to revisit this critique of the New York Times' Nate Cohn in the wake of the Walker implosion. If you remember, little over a month ago, Cohn made the argument that:

In the end, Mr. Trump almost certainly won’t win the Republican nomination; the rest of the party will consolidate around anyone else. He can influence the outcome only if his support costs another candidate more than others. But for now, he seems to be harming all candidates fairly equally.

This was never a convincing claim (at the time I called it "strained and convoluted"), and it has gotten even less defensible in the light of recent events, but that's not necessarily enough reason to dredge things up. There's a difference between keeping score and piling on (and the last post on Cohn was a bit on the harsh side).

I was leaning toward dropping the thread until I read Cohn's piece on the collapse of the Walker campaign and again saw things that bothered me. It was better than the Trump pieces, but it was improvement without progress.

[Yeah, I know, it sounds garbled but let me unpack the oxymoron. If the most recent outcome is better than the previous one, that's an improvement; if conditions change so that we can expect future outcomes to be better than previous outcomes, that's progress. (My blog, my definitions.)]

Over the past three months, Nate Cohn and many of his colleagues have not only failed to anticipate major developments; they've also made a string of predictions that haven't come to fruition. This wouldn't be worrisome if it were leading to a critical examination of the analogies and assumptions that led to these errors, or at least a little less self-assurance.

What we don't want to see is yet is yet another simple narrative presented as the explanation. Which brings us to..

Mr. Walker faltered so quickly because he simply was not skilled enough to navigate the competing pressures of appealing to the party’s establishment at the same time as arousing its base. It was much like the story of Rick Perry.

Though the entry of Donald Trump into the race made things harder for all the Republican candidates, Mr. Trump can’t be blamed entirely for Walker’s troubles. Mr. Walker was tied with Mr. Bush for second place in national polls heading into the first debate, long after Mr. Trump took a lead in those polls. By the time he dropped out, Mr. Walker had the support of less than one-half of 1 percent of Republican primary voters, according to the most recent CNN survey.

The Walker campaign — or perhaps the candidate personally — felt pressure from the rise of Mr. Trump on his right, especially once Mr. Walker started slipping a bit in the polls. This sort of pressure isn’t unusual and was inevitable — he would have felt it at some point, if not from Mr. Trump, then from Ben Carson or Ted Cruz.

Mr. Walker, to put it gently, did not handle this pressure well. His instinct was to move to the right as fast as possible at any point of vulnerability. He staked out a conservative position on birthright citizenship and a fringe position on considering a wall at the Canadian border. These moves alienated party elites and weren’t credible to conservative voters. He quickly reversed positions; in the end, he reassured no one.

First off there's the argument itself. It seems to be a reasonable, if not all that convincing, little story (not all that different than the one Nate Silver tells, but more an that in a minute) until you consider magnitude of the event being discussed. Walker did have notable missteps (to use Silver's term) but they were relatively minor. If we were talking about a campaign losing momentum or dropping a few points they might make sense as a possible cause, but we're talking about a complete implosion with a well-funded candidate going from near the top of the field to less than one percent support in shockingly little time.

The important part here is the speed of the collapse. Cohn himself was discussing Walker's weaknesses as a campaigner back in July, but he also said Walker had "plenty of time to assuage these concerns."

We now know Walker didn't. A month later, he would already be in free fall.

Scott Walker has sought to reassure jittery donors and other supporters this week that he can turn around a swift decline in the polls in Iowa and elsewhere by going on the attack and emphasizing his conservatism on key issues.

And, if you check the dates, you'll notice that the donors got jittery before Walker started talking about that Canadian wall.

Furthermore, probably Walker's most high-profile move to the right was the anti-labor position. I suppose this could have alienated the party elites (though we are talking about Koch brothers here) but if Scott Walker can't make an attack on unions credible to conservative voters, I honestly can't imagine anyone who could.

But the argument itself isn't what really bothers me. What's troubling here is the way that Cohn deals with the failure of his predictions. Even those with a good track record on the GOP race, such as Josh Marshall, admitted to being surprised at just how rapidly the collapse transpired. As a good rule of thumb, if you did not expect something, you should be careful about offering explanations about how it happened. As far as I can tell, Marshall and company are doing their best to follow this rule.

Cohn, by comparison, has perhaps the worst track record of any of the analysts I've been following when it comes to the GOP primary. After calling a premature end to the Trump surge a number of times and making the just-another-Herman-Cain analogy on no less than five separate occasions, he then made the previously mentioned claim about Trump not having any effect on the race.

Despite this, Cohn appears to be one of the most confident commentators when putting forward his theories about the causes of the implosion. The comparison to Nate Silver here is instructive. Silver actually opens his piece by acknowledging just how wrong he and his team were on Scott Walker. In addition to caution, Silver also offers a great deal more in terms of complexity and counter arguments. Cohn actually uses the word "simply" to describe his version of events.

But what's important here is not just the unwarranted confidence. We all have our moments. What merits attention is what Cohn's confidence in this situation tells us about his process. Anyone who works with data long enough will have occasion to see their models break down and their predictions go so far awry that they are no longer even directionally accurate.

When journalists looking in from the outside describe these disasters, they almost invariably use the phrase "go back and check your numbers," but in complex situations that is relatively seldom the source of the problem. More likely and far more difficult to catch are problems with robustness and modeling assumptions.

I realize I may be making too much of this, but there's a bigger issue here that has been bothering me for a long time. When Nate Cohn says this is what happened to Scott Walker, he is displaying a this-is-how-the-world-works tone and mindset that is very common in places like the Upshot, even when it is not at all appropriate. If you read carefully the work of journalists inspired by Nate Silver (though not so much Silver himself), you pick up the implicit belief that the standard methods and assumptions being employed are as true and as reliable as the laws of mathematics , that they have always worked and will always work.

This is a dangerous way to approach the social sciences, particularly when you start running into range of data issues (and between Trump, Carson, and the rise of the tea party, I think we are definitely in new territory now).

As mentioned before, I strongly suspect that the theory that Walker collapsed because his move to the right offended the elites and yet was not credible to the base is wrong, but I would be much more comfortable with it and would certainly have not written this post if it had been clearly framed as a theory.