The myth of orthogonality

One of the factors that contributed to the punched-in-the-gut feeling that so many people had immediately after the election was the seeming orthogonality of the data.

I'm using orthogonal in the broad rather than technical sense here (though I suspect both might apply) meaning to bring new information into the model. It wasn't just that the poll aggregators (with the partial exception of the outlier 538) were all telling us that the outcome was almost certain; we were also hearing exactly the same thing from pretty much everyone else, sources which supposedly had access to different information and were using a variety of approaches. A partial list included prediction markets, expert analyses, pseudo-exit polls (Slate's ill-fated, badly-thought-out Votcastr), and (from what we can infer) the consensus opinions within the campaigns themselves. All of these converged on exactly the same, completely wrong conclusion.

It is likely to take a great deal of hard work and deep digging to uncover exactly what went wrong here, but we can make some educated guesses:

The other data sources were never all that orthogonal (and possibly never all that good). For instance, even under ideal circumstances the predictive power of the markets was always overstated and overhyped, and presidential elections are nowhere near ideal circumstances.

To make matters worse, whatever orthogonality these other sources once brought to the table had faded to nothing by the time we got to this election. Between their early successes and the ludicrous amount of attention they received, the poll aggregators' predictions increasingly dominated conventional wisdom and became the only input (direct or indirect) that mattered for all the other “independent” sources of information.

I suspect that we reached the point where (if you'll forgive a clumsy phrase) prediction markets and the rest were anti-orthogonal. By providing the illusion of independent confirmation of the flawed polling data and likely voter models, they actually made it more difficult to bring new information in the system. It is entirely possible that better informed (or at least less misinformed) voters might have acted very differently, which suggests that the consequences of this particular failure may have been high indeed.

It's not what you'd call a pretty sound...

... but I'd still like to hear one in person.

Wheelharp



[Serious blogging to resume after the holidays.]

"As God as my witness..." is my second favorite Thanksgiving episode line [Repost]

If you watch this and you could swear you remember Johnny and Mr. Carlson discussing Pink Floyd, you're not imagining things. Hulu uses the DVD edit which cuts out almost all of the copyrighted music. .

As for my favorite line, it comes from the Buffy episode "Pangs" and it requires a bit of a set up (which is a pain because it makes it next to impossible to work into a conversation).

Buffy's luckless friend Xander had accidentally violated a native American grave yard and, in addition to freeing a vengeful spirit, was been cursed with all of the diseases Europeans brought to the Americas.

Spike: I just can't take all this mamby-pamby boo-hooing about the bloody Indians.
Willow: Uh, the preferred term is...
Spike: You won. All right? You came in and you killed them and you took their land. That's what conquering nations do. It's what Caesar did, and he's not goin' around saying, "I came, I conquered, I felt really bad about it." The history of the world is not people making friends. You had better weapons, and you massacred them. End of story.
Buffy: Well, I think the Spaniards actually did a lot of - Not that I don't like Spaniards.
Spike: Listen to you. How you gonna fight anyone with that attitude?
Willow: We don't wanna fight anyone.
Buffy: I just wanna have Thanksgiving.
Spike: Heh heh. Yeah... Good luck.
Willow: Well, if we could talk to him...
Spike: You exterminated his race. What could you possibly say that would make him feel better? It's kill or be killed here. Take your bloody pick.
Xander: Maybe it's the syphilis talking, but, some of that made sense.

This is Joseph

I hadn't been thinking about density like this, but here is a good point about LA:

LA suffers from the too-dense-but-not-quite-dense-enough problem (overall it's a very dense city, but with a kind of uniform density that is a bit difficult from a transportation perspective). 

I have generally hold that density is a pure good for transit, but it seems obvious that "destinations" would make things work a lot better than thousands of point to point connections.  The former (New York density) seems to make transit look super efficient.  But if you are relatively dense everywhere, that almost makes a cab/uber style of transit look efficient. 

In anticipation

Thanksgiving, 1905 from the incomparable Windsor McCay

from Mippyville.

Another entry in the Annals of Bad Ideas

This is Joseph

From Slate:

An anonymous elector told Politico that the House election that would result from a Trump Electoral College defeat “would immediately blow up into a political firestorm in the U.S." and be a positive step toward galvanizing the public’s support for ditching the college. Another, former Democratic National Committee Vice Chairwoman Polly Baca, said she’d prefer that the Electoral College return to the model outlined by Alexander Hamilton in the Federalist Papers and become an independent body informed but not governed by actual vote tallies.

I am especially surprised by the second suggestion here: just how does this help the problem?  You still have a potential difference between the popular vote and the electoral college.  But now the college can go completely rogue and decide the election.  How do we know that this version of the college, with what sound like unbound electors, will improve transparency.  The problem that people seem to worry about is the discordance between the electoral college and the popular vote -- how does this help? 

As for the first, the electoral college is currently in the constitution of the United States.  Last time I looked, it would take staggering levels of public support to pass a constitutional amendment.  Especially since any such amendment would obviously disadvantage whichever major party draws lots of support from rural and low population states.  I like the general idea of reforming this system, but I am unclear if continuing to destroy the norms of governance is going to be a good plan.

One may want to have them around at some point. 

One more for the lexicon: Curse of the First Model

This applies to a wide range of contexts, but, just to get the conversation started, let's say you're doing targeted marketing. You can come up with a fantastic mailing model, one that improves on the previous one in every conceivable way – – better response even if you mail somewhat deeper, more stable, and using data that are more reliable, cheaper, easier to work with – – and yet you will still get, at best, a lukewarm response from the executives. Invariably, you will be told something like this, "that's nice but we got so much more lift from the first model."

The trouble is that the improvement you see going from model to better model is almost always underwhelming compared to the improvement you see going from nothing to model. This curse can badly distort reputations and often leads to a kind of super Peter Principle where people are promoted to a level one step higher than their perceived level of competence which is much higher than their actual level of competence.

And no, we are not going to talk about the movie version

I have a pet theory that you can get some of the best insights into a period, not from the serious novels and plays of the era, but from good, successful popular art. You can learn a lot about a group by studying the stories that connected with them. You can also see how those people change over time

Case in point, in the Truman and Eisenhower administrations, it was remarkably easy to find Soviet bogeymen in movies and television. By the mid 60s, sympathetic and even lovable Russian characters were commonplace in movies and television. Think the Russians are coming, the Russians are coming, The Man from U.N.C.L.E., Chekov from Star Trek and many others.

Even James Bond changed with the times. In the 50s, he battled SMERSH, a fictionalized version of a real Soviet organization combined with elements of the KGB. In the 60s it was SPECTRE, a cartoonish terrorist network with a propensity for playing East against West.

I originally assumed that the change in attitude was a reaction to the Cuban Missile Crisis, but the more I look, the more it seems the shift started earlier.

Which takes us to Matt Helm. Donald Hamilton's Helm wasn't just the American answer to James Bond; he was the Fawcett Gold Medal answer. A contemporary of Travis McGee and part of a tradition that stretched back to the film noirs of the 40s and the pulps of the 20s and 30s. As Anthony Boucher pointed out, Hamilton brought the sensibility of Hammett to the novels. Helm was as unsentimental and clear-eyed as the Continental Op.

This exchange from the 1960 novel the Wrecking Crew illustrates that sensibility, and you'll notice that the tough-mindedness extends to the political as well. Helm doesn't claim to be fighting on the side of the angels nor does he demonize his opponent. Instead he explicitly equates his role and motives with those of the man he was sent to kill.

Donald Hamilton wasn't John LeCarre and this isn't The Spy who Came in from the Cold. The Helm novels are meant to be fast-moving adventure novels with great mass appeal. That makes the lack of jingoism all the more interesting.

Page 107

"If you got other orders," she said, "would you really – –"

I said irritably, "let's not go into the morality lecture, honey. I've heard it before."

"But it doesn't make sense!" She cried with sudden vigor. "You're a… an intelligent person. You're even kind of… kind of nice at times. And still you'd hunt down the human being like… like…" She drew a long breath. "Don't you realize that if this man Caselius is so evil and dangerous that he must be removed, there are other ways, legal ways… Can't you see that I resorting to violence, you just bring yourself down to his level, the level of the animals? Even if you should win that way, it wouldn't mean anything!"

There was a change in her attitude that puzzled me, a kind of honest indignation that was incongruous and disconcerting under the circumstances. A day earlier, a few hours earlier, I have spent some time trying to figure it out, but it was too late now.

There comes a time in every operation when the wheels are turning, the die is cast, the cards are dealt, if you please, and you got to go on as planned and hope for the best. I can name you names, too many of them, of men I've known – – and women, two – – who died because some last-minute piece of information made them try to pull a switcheroo after the ball had been snapped and the back field was in motion. When that point comes, to scramble the similes even further, you just take the phone off the hook and walk away from it. You don't want to hear what the guy at the other end of the line has to say. You've done your best, you've learned everything possible in the time at your disposal, and you don't want anymore dope on any part of the situation, because it's too late and you can't do anything about it, anyway.

I said, "That's kind of a funny speech from you, Lou. It seems to be kind of a set speech in these parts. Sarah Lundgren – – I think you've heard the name – – made it to, a few minutes before your Caselius put a nice accurate burst from a machine pistol into her face and chest."

I made an impatient gesture. "What the hell makes everybody feel so damn superior to this fellow Caselius? As far as I can make out, he's a bright, ruthless guy working like hell for his country, just like I'm a bright, ruthless guy working like hell for mine. His country doesn't happen to like my country. He's responsible for the deaths of a couple of people I'd rather have seen keep on living. I even got some sentimental objections to his methods. Therefore it's not going to grieve me deeply if I get orders to go ahead and make the touch.

"But as far as feeling superior to the guy, nuts! I'm perfectly happy to be on his level, doll. It's the level of a tough, intelligent, courageous man who could probably make a better living selling automobiles or insurance or whatever they sell in Russia, but who prefers to serve his country in the front lines, such as they are today. I don't hate him. I don't despise him. I don't look down upon him, as everybody else seems to, from some kind of a higher moral plane. I'm just prepared to kill him when and if I get the instructions to do so, whether it means anything or not. Meanwhile, I'd like to find out who he is."

With all this talk of moon bases and Mars missions

The unique conditions of the Post-war Era, tremendous optimism and prosperity balanced by Cold War anxiety got us to the moon. Space enthusiasts tend to overstate the part that the first element played and to overestimate the inspirational impact of the space race – the initiative was always controversial – but inspiration did play a role. Understanding that excitement is an important part of understanding what came next

The idea that conquest of the heavens was not just possible but eminent was greatly furthered by this...

From Wikipedia:

Man Will Conquer Space Soon! was the title of a famous series of 1950s magazine articles in Collier's detailing Wernher von Braun's plans for manned spaceflight. Edited by Cornelius Ryan, the individual articles were authored by such space notables of the time as Willy Ley, Fred Lawrence Whipple, Dr. Joseph Kaplan, Dr. Heinz Haber, and von Braun. The articles were illustrated with paintings and drawings by Chesley Bonestell, Fred Freeman, and Rolf Klep, some of the finest magazine illustrators of the time.

For more, check this out.

Cracked.com: " Why Hollywood Can Lose Billions & Still Make Terrible Movies"

One of these days we ought to do a deep dive into dysfunctional corporate decision-making, but for now I think the missing piece in the explanation of the green-lighting of seemingly doomed projects is the asymmetric risk associated with conventional versus unconventional decisions.

When everyone else is doing something, even when it is something that has failed badly and consistently in the past, you probably won't lose your job for doing the same thing. While your mother may not have been impressed by the "everybody else was doing it" defense, it is generally good enough to satisfy a Board of Directors.

Terrestrial superstation blogging – – MGM's the Works

[The over-the-air television industry continues to show remarkably strong and steady growth both in revenue and in number of stations (the last time I rescanned my television, I found well over 100). Press coverage has grown too, but at a far, far slower rate. The cynic in me might point to this as yet another data point in the argument that 21st-century reporters are only interested in stories that focus on the top quartile of the income distribution and have massive PR budgets behind them. Whatever the reason, there is remarkably little being written on the subject, so I thought it would be a good idea to do an occasional series of posts introducing some of the players.]


From Wikipedia:

The Works is an American digital broadcast television network that is owned by the MGM Television division of Metro-Goldwyn-Mayer.   …  Through its ownership by MGM, The Works is a sister network to This TV, a joint venture between MGM and Tribune Broadcasting which also focuses on films and classic television series from the 1950s to the 1990s and carries programming from The Works' corporate cousin MGM Television.

Compared to most studio--affiliated terrestrial superstations, the programming here is a bit of a hodgepodge, ranging from old movies to stand-up comedy shows to the odd sporting event like the Home Run Derby to HuffPost Live. This does not necessarily have to be a bad thing if the people running the channel know what they're doing, but no one at The Works appears to have put any thought whatsoever into pulling together the various strands. Nor has there been any apparent effort to come up with interesting and distinctive branding ideas.


The Works is also not at all insomniac friendly. While most of its peers program 24/7 (NBC/Universal's trainwreck COZItv -- which, for some reason, throws in a couple of hours of paid programming a day -- being the most notable exception), the Works runs late-night infomercials. I feel this is almost always penny-wise and pound-foolish. Late nights are a great time to build viewer loyalty, reinforce your brand, and just play around. To take an example from the world of cable, look at Cartoon Network and Adult Swim.

I do, however, have a couple of very nice things to say about the channel, One general and one specific.
Like all terrestrial superstations, MGM's the works is a good thing. For starters, you don't have to pay for it. More importantly(And this is the part I really, really like), you don't have to pay a cable company for it. I know I'm not alone in my feelings toward cable providers, phone companies, and satellite services. All of these industries have horrible reputations and long records of despicable business practices. This pattern of bad behavior is largely the result of operating mostly under monopoly or near-monopoly conditions. Even with the internet, you still have to deal with many of the same providers. For the moment, over-the-air television represents by far the healthiest competitive force in the field of live TV.

The second reason is specific to the Works. As you probably know, virtually every major piece of  popular art you can think of that is still under copyright is owned by a tiny handful of major players. Most of those companies appear to own or be in a relationship with at least one terrestrial superstation and these stations have been aggressively mining their owners' libraries. Furthermore, many of these stations are either operated by Weigel or modeled after their channels. As a result we have a lot of smart people who know what they're looking for digging through a fantastically rich collection of material.

Someone at the Works must've realized that MGM currently holds a great collection of major and minor gems of British cinema from the 30s through the 70s. A given week is likely to have multiple showings of something vintage from the likes of Alfred Hitchcock, David lean, or Noel Coward. There are also lots of not necessarily good, but interesting and hard-to-find films like Richard Lester's unique surrealist black comedy the Bed-sitting Room or First a Girl (the English remake of the German film Viktor und Viktoria which was in turn later remade by Blake Edwards as a vehicle for his wife Julie Andrews). I even saw a very nice print of Alfred Hitchcock's 1927 film the Lodger, nicely restored with an original soundtrack and tinting of selected done, I assume, to match the original.

On the plus side, it's an industry disruptor from a visionary CEO

I'm just going to post videos till I get my head around things

More thoughts on the economy

This is Joseph:

From the Guardian:

Median family income is lower now than it was 16 years ago, adjusted for inflation. Workers without college degrees – the old working class – have fallen furthest. Most economic gains, meanwhile, have gone to top. These gains have translated into political power to elicit bank bailouts, corporate subsidies, special tax loopholes, favorable trade deals and increasing market power without interference by anti-monopoly enforcement – all of which have further reduced wages and pulled up profits.

This is both a reasonable explanation for the rise of Trump, and it does not require any special appeal to racism or misogyny (not that both weren't present, but they would be very depressing as a complete explanation).  The key item is to think about ways to tackle these issues without introducing xenophobia.  Once the Democrats have decided on how to do this, the path back will be a lot clearer.

h/t: Mike

Talking Points Memo

This is Joseph

It might seem to be a very odd time to subscribe to Talking Points Memo; I did so on Tuesday, only after the subscriber drive and election were both pretty much over.  But the publication is considering taking an interesting direction.  One thing that seems clear to me is that it probably cannot hurt to put some real reporting into recent political shifts.  I have some suspicions about what happened, some of which are humbling.  But no matter what the cause, improved media depth of coverage cannot hurt matters, and may well really help.