What is the new interest in complexity?

One thing that I have been wondering a lot about lately is the move towards more and more complex ways of doing things that should be relatively simple.  Consider a few examples:

1. School choice and voucher systems.  This requires an informed parent to research options, see through marketing hype and balance factors like location versus performance.  How is this necessarily better than a system designed to just improve schools?
2. Medicare advantage programs.  This requires older adults to balance complex features of different plans and try to ensure that their provider is going to treat them well at a point of great personal stress.  How is having the government or the courts acting as a post-hoc check better than just have a simple system of insurance to begin with?
3. 401(k) and other defined contribution accounts.  Individual investors have enormous information deficits relative to instituitional investors.  Individual investors bear far higher levels of market risk and making the funds able to be withdrawn (even at a penalty) forces complex balancing decisions.  How is this better than an automatic pension plan like Social Security?

I totally understand that high information individuals may be able to optimize more clearly in a complex environment.  But the increase in transaction costs (needing to advertise, sorting through options) seems to suggest the median user will be worse off. 

Now this might be different if there was an open market in any of these examples.  But medicine is tightly regulated, we have laws saying that children must go to school, and anybody who was worked for an employer with a badly selected 401(k) knows that there is no free market alternative to shop your account to another employer.  It is not like a restaurant or a clothing store where the conditions for free markets will end up making it easy to find what you want.  But you can't switch health care providers or schools based on the sale of the week. 

So why do we want to make these things harder and harder to understand or engage with? 

Even Jaime Escalante wasn't a Jaime Escalante

A quick follow-up to Joseph's last post, particularly his use of the late Jaime Escalante as an example.

Perhaps the central assumption of the reform movement is the belief that the best way to fix education is by identifying and and retaining great teachers (generally defined as those who can dramatically raise students' test scores). Quite rightly, Escalante is the first name that comes to most people's mind when they hear about this, but his career suggests that being a great teacher is a necessary but not sufficient condition for getting great results.

Following Mark's link on education

As a follow-up to Mark I wanted to specically call out some of the pieces of Felix Salmon's piece on test scores and education. 

Instead, reformers are rushing to use this data as a quantitative performance-review tool, something which can get you a raise or which can even get you fired. And by so doing, they’re turning it from something potentially extremely useful, into a bone of contention between teachers and managers, and a metric to be gamed and maximized.
 

When all decisions on based on a single score, you incent behavior which maximizes the score and minimize additional focus.  Felix makes an interesting point that if you used this data to provide coaching and feedback then it could actually be really useful.  Teachers would still want students to do well on the test (it is much, much nicer to talk to your principal about how generally well your students are doing than to get coaching on how to try and shore up a weak point). 

I also think that this point was really sharp:

School reformers in general, it seems to me, tend to be obsessed with the idea of Good Teachers and Bad Teachers, as though the quality of the education a kid gets in any given classroom is somehow both predictable and innate to the teacher. And yes, at the extremes, there are a few great inspirational teachers who we all remember decades later, and a few dreadful ones who had no idea what they were talking about and who had no control of their classes. But frankly, you don’t need student surveys to identify those outliers. And the fact is that schools are much more than just the sum of their parts: that’s one of the reasons that reformers love to talk about excellent principals who can turn schools around.

He is very cleverly and accurately pointing out a form of equivocation that is being used here.  There are extreme examples, but they were never the problem in terms of identification.  There are some odd employment rules in some places that made acting on this knowledge awkward, but very few people saw these as being good policy.  The real use of these tests to to try and break apart the middle of the distribution.  But, by definition, the gain in the middle of the distribution is much less than the difference between exceptional and abysmal.  You are not taking Jaime Escalante versus an incompetent as your contrast.  You are taking pretty good versus very good as your contrast, and thus setting things up for a life event to move people back and forth in the distribution.  Your child gets ill, you are more tired and work so you lsoe your job because you slip below the median.  No wonder teachers are suspicious of such metrics. 

Data is good but one of the lessons of the MBA approach to management is that not everything can be broken down into numbers on a spreadsheet to be maximized.  I fear we'll figure that out, sooner or later. 

Essential Felix

It's a busy week so I don't have the time to give this the build-up it deserves, but if you've been following the education reform debate, you need to read this.

The main reason epidemiology is hard... urgency

There's a point I should have emphasized in my previous post about this news story

Researchers have long documented that the most educated Americans were making the biggest gains in life expectancy, but now they say mortality data show that life spans for some of the least educated Americans are actually contracting. Four studies in recent years identified modest declines, but a new one that looks separately at Americans lacking a high school diploma found disturbingly sharp drops in life expectancy for whites in this group. Experts not involved in the new research said its findings were persuasive.

I commented that cohort effects could complicate things in a study like this. What I should have made clear was that, even with these complications:

1.  These numbers are horrifying;

2. Even if the effects are less dramatic than what the most recent study indicated, there is considerable evidence that life expectancies are getting worse for the most disadvantaged people in our country.

I'm primarily a marketing statistician. I hardly ever make life and death decisions. Epidemiologists play for higher stakes. I believe it's healthy for outside statisticians like me to point out concerns with epidemiological research, but we need to remember that researchers inside the field don't have the option of waiting for a perfect data set. Their findings have a huge impact on the health of millions of people and those millions can't wait for perfect.

More on the whole "Epidemiology is Hard" thing

There's a new study that's been getting a lot of press:

For generations of Americans, it was a given that children would live longer than their parents. But there is now mounting evidence that this enduring trend has reversed itself for the country’s least-educated whites, an increasingly troubled group whose life expectancy has fallen by four years since 1990.
Researchers have long documented that the most educated Americans were making the biggest gains in life expectancy, but now they say mortality data show that life spans for some of the least educated Americans are actually contracting. Four studies in recent years identified modest declines, but a new one that looks separately at Americans lacking a high school diploma found disturbingly sharp drops in life expectancy for whites in this group. Experts not involved in the new research said its findings were persuasive.

The reasons for the decline remain unclear, but researchers offered possible explanations, including a spike in prescription drug overdoses among young whites, higher rates of smoking among less educated white women, rising obesity, and a steady increase in the number of the least educated Americans who lack health insurance.

The steepest declines were for white women without a high school diploma, who lost five years of life between 1990 and 2008, said S. Jay Olshansky, a public health professor at the University of Illinois at Chicago and the lead investigator on the study, published last month in Health Affairs. By 2008, life expectancy for black women without a high school diploma had surpassed that of white women of the same education level, the study found.

White men lacking a high school diploma lost three years of life. Life expectancy for both blacks and Hispanics of the same education level rose, the data showed. But blacks over all do not live as long as whites, while Hispanics live longer than both whites and blacks. The decline among the least educated non-Hispanic whites, who make up a shrinking share of the population, widened an already troubling gap. The latest estimate shows life expectancy for white women without a high school diploma was 73.5 years, compared with 83.9 years for white women with a college degree or more. For white men, the gap was even bigger: 67.5 years for the least educated white men compared with 80.4 for those with a college degree or better.
...
The decline among the least educated non-Hispanic whites, who make up a shrinking share of the population, widened an already troubling gap. The latest estimate shows life expectancy for white women without a high school diploma was 73.5 years, compared with 83.9 years for white women with a college degree or more. For white men, the gap was even bigger: 67.5 years for the least educated white men compared with 80.4 for those with a college degree or better.

This is, unquestionably, a troubling finding and I have every reason to believe that the researchers did a responsible job. None the less, this part still troubles me:

Researchers said they were baffled by the magnitude of the drop. Some cautioned that the results could be overstated because Americans without a high school diploma — about 12 percent of the population, down from about 22 percent in 1990, according to the Census Bureau — were a shrinking group that was now more likely to be disadvantaged in ways besides education, compared with past generations.

Professor Olshansky agreed that the group was now smaller, but said the magnitude of the drop in life expectancy was still a measure of deterioration. “The good news is that there are fewer people in this group,” he said. “The bad news is that those who are in it are dying more quickly.”

Dying at the age of seventy in 1990 would mean you were born in 1920. Dying at the age of sixty-seven in 2008 would mean you were born in 1941. If you were to build a model to predict whether someone born in in 1920 would finish high school, it would certainly look different than a model predicting the same thing for someone born in 1941. We know this, for one reason, because the target variable was much less frequent for the second group.

Of course, we're talking about averages and distributions and that can complicate things in a number of ways (most of which don't make it into a newspaper account of a research paper), but no matter how you slice it, what it meant to have a high school diploma changed greatly from the early to the middle part of the Twentieth Century, and that change is very difficult to control for.

I think it's generally a good, conservative rule to assume that when the relative size of a segment of the population drops dramatically, the composition of the segment is likely to shift. Thus a sharp increase in a behavior in these groups' behavior may simply be a result of the people who didn't show the behavior not being in the group any longer.

I'm not saying that this is the case here. I'm just saying that this sort of analysis makes me  nervous.

(I have the same problem with dropping poll responses, but more on that later

This does not seem to be a hoax

From Marketplace:

But the course of true investing never did run smooth, and there are some traders who look to the stars to tell them what to do. Financial astrologers like Karen Starich say traders know they're up against a lot of rich, smart people.

"They want to have that edge," she says. "They want to know what the future is."

Starich chargest $237 annually for her newsletter, which 300 traders subscribe to for news of what will happen to the stock prices of companies, or even bigger, to the Federal Reserve. She sees dark times ahead in the Fed's horoscope.

"They now have Saturn squared to Neptune, which is really bankruptcy," Starich explains.

Neptune represents money. But when Saturn shows up in a chart, it indicates restriction. So for the Fed, that means the "fiscal cliff is here, and there’s no place to go except to print more money or unravel these financial institutions," Starich says.

Of course, a lot of Wall Street traders, and others, don't want it to be known that they're relying on anything other than their own talent. Arch Crawford, a financial astrologer who actually got his start on Wall Street as a stock analyst at Merrill Lynch, recalls one subscriber asking for his newsletter in "brown paper wrappers."

Crawford warns his 2,000 subscribers particularly against the dangers of Mercury in retrograde, a time when the planet appears to be going in reverse across the sky. The phenomenon, which happens three times or more a year, indicates a month when communications will be screwed up. He warns his subscribers never to start anything new during that time. He points to the fact that Knight Capital launched a new software program in August, when Mercury was in retrograde, and the brokerage firm nearly went out of business. He also notes that most major market glitches have happened while Mercury was in retrograde.

LA billboards -- another n = 1 hypothesis

I have absolutely nothing more than vague impressions to back this up but it seems that a disproportionate share of the billboards of in this town, particularly around Burbank and Hollywood, are dedicated to TV shows. Of course, LA is a trend setter and a lot of media writers are based here, but billboards seem an inefficient way of targeting a relatively few influential players.

I suspect that (if I'm right about the over-representation) there's a bigger factor that has nothing to do with maximizing corporate profits. Instead the objective is to maximize the profile and reputation of the executives running the networks. The target audience is colleagues and acquaintances and the purpose is to create the impression that the executives are successful and influential.

William Goldman once observed that studio executives would rather make big budget films even when low budget alternatives made more business sense because, even if the big films lost money, they still gave you something to talk about at parties. ("I just greenlit the new Hanks/Roberts project. How about you?") After reading that, a number of why-did-they-make-that pictures were suddenly more explicable.

Lots of business decisions make more sense when you apply this cocktail-party standard (including the majority of sports sponsorships), but I wonder if anyone has seriously tried to study and quantify the extent to which home industries of a region are over-represented in local advertising. If not and if anyone's interested, I might have some design and metric ideas.

Back on the down ticket question

As mentioned before, one of the most interesting aspects of this election is the way the different races interact.

Thomas Ferraro has a good report on the topic:

It is axiomatic that a strong presidential nominee can boost the chances for other party candidates, particularly those in close U.S. Senate races.

But the presidential candidate can also hurt those farther down the ticket.

While a single comment might not alter particular races, a lagging campaign could. Republican Representative Steven LaTourette of Ohio, who is retiring from Congress, defended Romney's remarks, but said they "don't help in swing districts like mine." "People were ready to throw Obama over, like dumping a boyfriend, and were ready to be courted by a new boyfriend," he said. "But now they're having second thoughts," LaTourette said.

Republican Representative Tom Cole of Oklahoma called Romney's remarks "an unfortunate choice of words," but predicted the comments would be "a one- or two-day story."

"The election is going to turn on the economy," Cole said. University of Virginia political scientist Larry Sabato, whose "Crystal Ball" blog closely tracks congressional races, said Romney's performance would be particularly influential in Senate races in Virginia, Connecticut, Montana, North Dakota and Florida.

"Scott Brown can't survive much more undertow in Massachusetts," he said. George Allen, the Republican Senate contender in Virginia, "depends on a Romney win," Sabato added.

"As I go through the states, I'd say Romney's performance will help to determine most of the close Senate contests," he said in an email interview.

"It's going to be very difficult for Republicans to take over the Senate if Romney doesn't capture the White House. That's a different evaluation than a year ago when the GOP looked to be a good bet to grab the Senate."

Why is investment in human capital out of favor?

David Brooks:

The final thing the comment suggests is that Romney knows nothing about ambition and motivation. The formula he sketches is this: People who are forced to make it on their own have drive. People who receive benefits have dependency.

But, of course, no middle-class parent acts as if this is true. Middle-class parents don’t deprive their children of benefits so they can learn to struggle on their own. They shower benefits on their children to give them more opportunities — so they can play travel sports, go on foreign trips and develop more skills.

People are motivated when they feel competent. They are motivated when they have more opportunities. Ambition is fired by possibility, not by deprivation, as a tour through the world’s poorest regions makes clear.

 

I got this quote via Aaron Carroll.  One thing that seems to have really changed in the public mind over the past 5 years is the simple relationship between investment in human capital and improved outcomes.  Putting aside the Randian fantasy of super-men who can do it all on their own, all of us are dependent on others to a greater or lesser extent.  Hard work may well make opportunity into success but all of the hard work in the world doesn't help in the absence of opportunity.  Just ask any coal miner or subsistence farmer . . . 

Another recommend (this time not blind)

Dana Goldstein has some smart observations on Chicago and the teacher testing debate.

Another reason observational epidemiology is hard

John D Cook:

And yet behind every complex set of rules is a paper showing that it outperforms simple rules, under conditions of its author’s choosing. That is, the person proposing the complex model picks the scenarios for comparison. Unfortunately, the world throws at us scenarios not of our choosing. Simpler methods may perform better when model assumptions are violated. And model assumptions are always violated, at least to some extent.
 

 One of the hardest things with simulation studies is that we get to develop our own set of assumptions.  So we actually know how to correctly model the phenomenon of interest. 

The problem is that we usually do not know how much error is introduced when the complex (and often non-linear) model fails.  On the other hand, it is amazing how far one can get with a clear set of rules of the thumb. 

I wonder if it would be better if we had a different person test the model than the one who proposed it? 

Important if true

In the Nineteenth Century, news stories sometimes carried the disclaimer "Important if true." It was used for news that couldn't be confirmed but was too big to leave unreported. That line always struck me as a perfect representation of one of the three fundamental tensions of journalism, the one between getting it right and getting it first (the other two are between these goals and making the story interesting).

In a follow-up to a discussion about the way the press corp handled the 2000 election, I claimed that the culture of journalism was placing increasingly less value on accuracy. To support the accusation, I listed some examples from recent discussions, but there is a more direct and damning way to make the point.

Viewed in absolute terms the level of accuracy in current journalism is certainly not what it could be and is arguably in decline, but what about relative to cost? One hundred years ago, the time required to check a story was substantial. Sources were difficult to contact, reference materials had to be accessed in person and were often unindexed, and there was simply less recorded information. This time requirement translated to both labor and opportunity costs.

Over the years (particularly the last twenty), these costs have dropped sharply. The time required to get a story right has decreased by at least a couple of orders of magnitude. If we hold the value of accuracy constant, we would expect to see factual errors all but disappear. Instead we're seeing something between "as bad as ever" and "even worse." That means the value journalism as an industry puts on accuracy must have dropped as or more sharply than the costs of fact-checking did.

Think of it as Moore's law of diminishing journalistic quality.

Blind recommendation

I haven't got a chance to listen to more than the first couple of minutes, but I'm still prepared to recommend This American Life's latest episode, Back to School. That's not to say I expect to agree with everything I hear, just that I expect to leave better informed than I came in.

A culture of bad journalism

Eight years ago, in the bastions of the "liberal media" that were supposed to love Gore—The New York Times, The Washington Post, The Boston Globe, CNN—he was variously described as "repellent," "delusional," a vote-rigger, a man who "lies like a rug," "Pinocchio." Eric Pooley, who covered him for Time magazine, says, "He brought out the creative-writing student in so many reporters.… Everybody kind of let loose on the guy."

Evgenia Peretz

A bit more on Gore and the press. In the last post discussed the press's treatment of Gore, the way various forces caused many (if not most) of the country's most respected and best paid journalists to converge on a set of highly biased and unprofessional actions. I still think that's a good way to approach the story (and I wish someone who actually knew something about social psych would pick this up and run with it), but there were at least a couple of other factors that acerbated the situation.

On Hardball, Chris Matthews stated: "Al Gore, he's the one who said he created the Internet. He's the one who put out the word that he was the subject or the role model for Love Story, that he pointed the country's attention to Love Canal. He stuck himself into that story." Matthews concluded: "Gore got himself in those problem areas by vanity and showing off and trying to make himself cool."

from Media Matters 2007

These three points were probably the most frequently cited examples of Gore's flawed character. They were also all false. Moreover, they had been debunked almost immediately. And yet they proved impervious to repeated attempts at correction.

The problem was partly that many of the journalists wanted to believe the story, but there was also a larger shift in the culture of journalism away from factual accuracy. This shift has continued and, if anything, gotten worse. My favorite recent example is the spendthrift Spaniards.

From an earlier post:

In yesterday's NYT, Rachel Donadio had a report on Italy that included this sentence:

Germany has adamantly opposed what it sees as rewarding the bad behavior of southern rim countries like Italy, Greece, Spain and Portugal, which amassed high public debts and where tax evasion is rampant.

Except, of course, they didn't. Dean Baker (who first caught this) debunks:

Actually, of this group only Greece was consistently experiencing a rise in its debt to GDP ratio. In Portugal there was some increase in the debt to GDP ratio in the years prior to the recession, but Italy's debt to GDP ratio actually had been trending downward since 2000. Spain was running budget surpluses and had a considerably lower debt to GDP ratio than Germany.

It's not just that the NYT didn't bother to check these facts; it's that they had been debunked repeatedly in numerous places including this column printed less than a month ago in, you guessed it, the New York Times.

For various reasons, accuracy is simply not a priority.

Now add in the embrace of memes and collective narrative. Narratives are useful for organizing ideas but they're dangerous as well. As Steven Kloves put it in Wonder Boys, narratives are about what you leave out, in other words, they are built on confirmation bias. By accepting the idea that the press can and should converge on a collective narrative, we are giving journalists permission to leave out important fact. We're also making it easier for interested parties to manipulate the news.

Put these three together, an easily manipulated press corps, a disinterest in factual accuracy and an acceptance of convergent and bad journalism becomes almost inevitable.

(Just to be clear, I'm not saying there aren't any good journalists out there; I'm saying that they're good despite the culture of their profession.)

Andrew Gelman forces me to read Bob Somerby

"Gore elicited in us the childish urge to poke a stick in the eye of the smarty-pants"

Margaret Carlson

I was going to write a reply to Andrew Gelman's recent criticism of Paul Krugman but since everybody beat me to it, I'll limit myself to this post script

P.P.P.S. Maybe the zillion commenters who disagree with me here have a point! I still find it a bit of a stretch for people to claim that reporters’ personal likes/dislikes would have more of an effect on coverage than reporters’ ideologies and partisanship, but I can see the reasoning, which I think roughly goes like this: journalists are trained to not let their partisanship get in the way of their reporting, but they don’t have that same constraint with respect to personal like/dislike. Thus a liberal Democratic reporter who personally liked Bush and disliked Gore might slant the news toward Bush and even feel good about such a slant in that it represents a bending-over-backwards to not simply follow the partisan cue.

As noted, I remain skeptical of this story—-I’d think that, when it comes to a national election, partisanship would trump personality—-but it is a coherent argument, supported by data. Which satisfies the request, posed at the top of this post: “I’d like to see Paul Krugman’s evidence for this.”

There are some potentially interesting side questions here about the cultural differences between the sides of the spectrum ( "I am not a member of an organized political party. I am a Democrat."), but time is limited so I'll jump straight to the Cialdini.

If you'll flip to the liking, reciprocation, consistency, and social proof chapters of Influence, you'll find lots of evidence to support this story, much of it from peer reviewed papers. Here are some of the relevant points.

Many people have noted Bush was capable of considerable charm.

Many influential reporters had a long standing and well documented dislike of Gore.

The Washington press corp is small and cliquish, prone to convergent behavior.

Because of the image of a liberal bias and because the GOP is known for pushing back, there are generally fewer consequences for a story that puts a Democrat in a bad light. Admitting you've backed down tends to cause cognitive dissonance which is resolved by convincing yourself you meant what you wrote.

The written word is particularly noted for affecting attitudes. People who put beliefs in writing are much more likely to embrace those beliefs.

Bush literally wined and dined the press corp. (Google "lobster ravioli" and Bush). This kind of gift is big enough to make an impression but not large enough that the reporters could justify a lapse in ethics (if I give you a million dollars to do something bad, it is easier to justify and creates less dissonance). Once again we get reciprocation, cognitive dissonance, modification of attitude.

Finally, we know that people tend to greatly underestimate how easily they can be influenced by any one of these things, but it's when they start reinforcing each other that you can get people to do truly startling things.

So, we have a scenario right out of a social psych book, fairly well documented examples of biased coverage, and an election so close that a major change in the tone of coverage could very probably have changed the result.

I think this one goes to Krugman.

Education: Chicago edition

The teacher work action in Chicago is bringing up some strong feelings across many different bloggers.  I think there is a lot to say about this issue (in general) but I want to focus on one angle for the moment.  Here is a comment from Matt Yglesias:

If CTU members get what they want, that's not coming out of the pocket of "the bosses" it's coming out of the pocket of the people who work at charter schools or the people who pay taxes in Chicago.
 

Now, to be fair, Matt has some follow up posts that reflect a more nuanced view of this dispute.  But this point was seized on by Eric Loomis:

It’s these experiences that make me absolutely furious when Dylan Matthews and Matt Yglesias and Jacob Weisberg and other so-called liberals attack Chicago teachers by openly rooting from Rahm Emanuel to crush them or undermine them by warning readers about the effect of paying teachers on taxpayers. I don’t really know any of them personally. But I doubt any of them went to a public school, nor has much of the liberal punditry. And if they have, it’s almost certainly not one serving working-class communities like areas of Chicago or even Springfield. They can sit in their nice New York or Washington offices and attend retreats in baronial mansions like Slate held earlier this week and fret about the taxpayers and shame the teachers into thinking about the children all they want. They would never send their own children to the schools about which they pontificate. They have no idea what they are talking about.
 

No I don't want to discuss whether teacher pay in Chicago is sensible or not (Matt Yglesias defends the current levels here).  What I find a lot more interesting is the whole question of mixed system (with public and private options co-existing).  As a younger person, I often asked the question of why Canada generally made private medical services illegal (they have definitely relaxed the rules since then).  After all, why should be ban a person who wants to spend money on non-evidence based procedure or get faster service from spending cash to do so?  We do not ban pet rocks or other products of limited use. 

The answer was always that if there was a parallel "pay system" then the elite would attack the public option (as they would use the higher end options almost exclusively).  The net result would not just be a two tier system, but a two tier system that was actually worse than a designed one (as the lower tier would be formed by a series of constant cuts and the result would be worse than a planned low service level).  At the time, I did not see this as a likely outcome. 

But linking Chicago teacher pay increases to less resources for charter schools (even implicitly) makes this a much more credible concern. 

More down-ticket thoughts

You've probably seen this:

 Rep. Paul Ryan, Mitt Romney's running mate, will begin airing ads defending his House seat in Wisconsin, the Associated Press reports. The ads will begin airing Wednesday.

For the record, there's nothing wrong with (or even unusual about) trying to hang on to a seat while on a national ticket, nor would I read this to mean that Ryan is running for the exit in response to a drop in the polls as one blogger put it.

I do, however, wonder if this might be one of the down-ticket effects that I speculated about earlier. When the Romney campaign pulled out of Wisconsin, did Ryan feel the need to move back in, even though he knew the move wouldn't look good?

The difference between Mark Twain and Dean Baquet...

Twain got the joke.

From Politico via Waldman:

Newly available CIA records obtained by Judicial Watch, the conservative watchdog group, reveal that New York Times reporter Mark Mazzetti forwarded an advance copy of a Maureen Dowd column to a CIA spokesperson — a practice that is widely frowned upon within the industry.
...
New York Times Managing Editor Dean Baquet called POLITICO to explain the situation, but provided little clarity, saying he could not go into detail on the issue because it was an intelligence matter.
...

"The optics aren't what they look like," he went on. 

In addition to the oxymoronic aspect, there's also the innately comic implication that a Maureen Dowd column could actually contain useful information.

p.s. I was originally going to post this earlier but I decided I needed to spread out the snideness.

Nate Silver's new book

Nate Silver has a new book that examines the state of predictive modeling in the age of big data. Here's an excerpt. If you need more recommendation than that, you haven't been paying attention.

These meteorologists are dealing with a small fraction of the 2.5 quintillion bytes of information that, I.B.M. estimates, we generate each day. That’s the equivalent of the entire printed collection of the Library of Congress about three times per second. Google now accesses more than 20 billion Web pages a day; the processing speed of an iPad rivals that of last generation’s most powerful supercomputers. All that information ought to help us plan our lives and profitably predict the world’s course. In 2008, Chris Anderson, the editor of Wired magazine, wrote optimistically of the era of Big Data. So voluminous were our databases and so powerful were our computers, he claimed, that there was no longer much need for theory, or even the scientific method. At the time, it was hard to disagree.

But if prediction is the truest way to put our information to the test, we have not scored well. In November 2007, economists in the Survey of Professional Forecasters — examining some 45,000 economic-data series — foresaw less than a 1-in-500 chance of an economic meltdown as severe as the one that would begin one month later. Attempts to predict earthquakes have continued to envisage disasters that never happened and failed to prepare us for those, like the 2011 disaster in Japan, that did.

The one area in which our predictions are making extraordinary progress, however, is perhaps the most unlikely field. Jim Hoke, a director with 32 years experience at the National Weather Service, has heard all the jokes about weather forecasting, like Larry David’s jab on “Curb Your Enthusiasm” that weathermen merely forecast rain to keep everyone else off the golf course. And to be sure, these slick-haired and/or short-skirted local weather forecasters are sometimes wrong. A study of TV meteorologists in Kansas City found that when they said there was a 100 percent chance of rain, it failed to rain at all one-third of the time.

But watching the local news is not the best way to assess the growing accuracy of forecasting (more on this later). It’s better to take the long view. In 1972, the service’s high-temperature forecast missed by an average of six degrees when made three days in advance. Now it’s down to three degrees. More stunning, in 1940, the chance of an American being killed by lightning was about 1 in 400,000. Today it’s 1 in 11 million. This is partly because of changes in living patterns (more of our work is done indoors), but it’s also because better weather forecasts have helped us prepare.

Perhaps the most impressive gains have been in hurricane forecasting. Just 25 years ago, when the National Hurricane Center tried to predict where a hurricane would hit three days in advance of landfall, it missed by an average of 350 miles. If Hurricane Isaac , which made its unpredictable path through the Gulf of Mexico last month, had occurred in the late 1980s, the center might have projected landfall anywhere from Houston to Tallahassee, canceling untold thousands of business deals, flights and picnics in between — and damaging its reputation when the hurricane zeroed in hundreds of miles away. Now the average miss is only about 100 miles.

Why are weather forecasters succeeding when other predictors fail? It’s because long ago they came to accept the imperfections in their knowledge. That helped them understand that even the most sophisticated computers, combing through seemingly limitless data, are painfully ill equipped to predict something as dynamic as weather all by themselves. So as fields like economics began relying more on Big Data, meteorologists recognized that data on its own isn’t enough.

Mark Thoma and the hothouse flowers of the right

In a post worth revisiting, Mark Thoma's head finally explodes at the sight of one-too-many delicate Republicans:

But really, what is it with Republicans and their hurt feelings? They tell us that the CEOs of major corporations stopped investing, stopped maximizing profit for their investors because the president hasn't honored them enough. They'll show him! -- all the while losing money for their investors? Republicans complain endlessly about the debt, it was a theme of their convention, but given a chance to do something about it they walk away because the president didn't treat them exactly as they expected and demanded? Apparently, their feelings got in the way. They show no respect to the president whatsoever -- quite the opposite -- and then break off negotiations they believe are crucial to the future of the country because he didn't show them the respect they think they deserve? Cry me a river (well, everyone but Boehner).

I'm surprised it took him this long. I lost it back in February when Amity Shales wrote this:

Obama wants to reward companies that create jobs here in the United States. One of the carrots is a tax credit for companies that move operations back here. Another would double tax breaks for high-tech factories making products here.

These are juicy carrots. But the sticks put forward by Obama are hefty. The president wants to eliminate a tax break for moving expenses when a company ships operations overseas. He also wants to close a tax loophole that allows companies to move some types of profits to overseas tax shelters.

The president figures that businesses will tolerate the pain of the sticks for the reward of the carrots. He thinks if he pokes the stick in one corner, they'll hop over to the corner where the carrots are.

But the trouble with this argument is that the U.S. economy is not a rabbit cage. And business people -- entrepreneurs especially -- don't respond well to prods from a stick. Any stick. If they get a glimpse of the rod, they'll leap away for sure -- but it might just be to somewhere outside the United States. Our cage. And the carrots of cheaper labor there overseas might even be tastier.

Maybe the president is forgetting the goal, which is making the economy grow faster. Enough carrots, and businesses will grow. And they'll create jobs. But pick up even just a few sticks, and you won't get recovery. Instead, we'll all be looking at an empty cage and asking: Where are the rabbits?

Here was my reaction at the time:

Putting aside the argument that eliminating "a tax break for moving expenses when a company ships operations overseas" will encourage companies to ship operations overseas (is there a paragraph missing somewhere?), what caught my eye was the way Shales tortures this poor metaphor.

It doesn't help that the proverbial carrots and sticks were used to motivate proverbial mules and other large and stubborn beasts of burden. As an old country boy, I can tell you that getting big animals to go where they don't want to go is a challenge. I haven't had that much experience with bunnies, but I have to think it's a bit less daunting. I don't even believe I'd need a stick.

But Shales' odd allegorical choice is in keeping with the even odder dichotomy in the way conservative rhetoric has come to treat entrepreneurs and business leaders. Half the time they're bold and decisive figures, the spiritual descendants of our frontier forefathers; the rest of the time they are as delicate as a hothouse flower and as timid as a woodland creature (like, for example, a rabbit).

Shales has entrepreneurs leaping away at just "a glimpse" of a rod (and given that she describes closing a couple of tax loopholes as "hefty" penalty, it's fair to say that she really does mean it when she says any stick). Other conservative commentators have speculated that business leaders are slow to invest because they can't deal with the uncertainty caused by a possible return to Clinton era tax rates. We've even heard some argue that the recovery was slowed because the president keeps saying hurtful things about bankers and CEOs.

It's a bit like Dr. Jekyll and Mr. Hyde but with John Galt and Elmo.

Snark aside, what's troubling here is both the persistence and influence of the poor-little-Randian argument. High-ranking politicians and respected journalists continue to use it to support policies with the potential to do huge damage to the economy while the rest of us continue to listen to them as if they were saying something sensible.

More non-snarky questions

This post by Jonathan Chait got me thinking about something that's been in the back of my head for a while:

Romney is targeting eight states: Nevada, Colorado, Iowa, Ohio, Florida, North Carolina, Virginia, and New Hampshire. No Wisconsin, Michigan, or Pennsylvania. This is surely not because Romney is husbanding scarce cash. Campaign aides also told Fox News yesterday that they basically have so much money they have to come up with ways to get it out the door, Brewster’s Millions–style, before election day. (“We have $100 million we've just raised. If you look at our burn rate to date and our cash on hand, there's not much more we can spend on infrastructure. So we've got to start spending our general election funds in a big way, because you know what the value of that money is on the day after the election? Zero.”) And it’s probably not because they want outside super-PACs to spend in those states, either — they can’t legally coordinate, and the super-PACs will take their cues from the Romney campaign about where to fight. (The GOP super-PACs have already pulled out of Michigan and Pennsylvania.)

The reason this looks worrisome for Romney is that he’s pursuing an electoral-college strategy that requires him nearly to run the table of competitive states. The states where Romney is not competing (and which aren’t obviously Republican, either) add up to 247 electoral votes. The eight states where Romney is competing add up to a neat 100 electoral votes, of which Romney needs 79 and Obama just 23. If you play with the electoral possibilities, you can see that this would mean Obama could win with Florida alone or Ohio plus a small state or Virginia plus a couple small states, and so on.

Unless I’m missing something badly here, Romney needs either a significant national shift his way — possibly from the debates or some other news event — or else to hope that his advertising advantage is potent enough to move the dial in almost every swing state in which he’s competing.

We could debate the tactical effect this is likely to have on the election, but what about the elections? What are the down-ticket effects of a move like this? Joseph and I had a conversation on the subject earlier today, but being the rankest Monday morning quarterbacks on the issue, all we could come up with were questions:

This has to make news in the affected states. In a race that seems to be much more about turnout than about persuading undecideds, doesn't this make Romney supporters less likely to show up and vote for the man? (and if they don't show up for Romney...)

If this effect is significant, has the GOP decided these states aren't worth the effort?

Or is the Romney campaign simply focused on its own interests here?

More on Andrew Gelman's tobacco post

Andrew Gelman's recent post on distinguished researchers who did less than distinguished work for the tobacco industry reminded me that I've been meaning to do some posts on Robert B. Cialdini's Influence (either the textbook or the mass market edition. They're both pretty much interchangeable).

For those not familiar with the book, Cialdini takes some well-established principles of influence such as the impulse to reciprocate and shows how these effects can be seen in psychological studies, historical anecdotes, news stories, everyday incidents and, most famously, marketing campaigns, then he wraps up by putting things into an evolutionary psych context that, God help me, actually appeals to common sense.

There's a lot of cool stuff in Cialdini's book (just the part about the "brainwashing" of Korean War POWs is worth the price of admission), but the relevant points for the Gelman post are:

1. When you give a researcher money while nominally refusing to dictate results (which, I suspect is how this normally works), you create a sense of obligation. This leads to cognitive dissonance -- the researchers wants to see him or herself as honest but at the same time wants to repay the company, which can only be done through bad research. The dissonance is often resolved by adjustment of personal beliefs (the researcher convinces him or herself that the research really does back up the company's position).

2. People tend to underestimate how much and how easily they can be influenced. Doctors insist that small gifts from drug companies don't influence them despite numerous studies that show the technique to be highly effective.

3. Subsidized research is dangerous.

I'm working from memory here and not doing Cialdini justice. He has tons of supporting evidence and numerous persuasive examples of these phenomena. Fascinating book, particularly for anyone with a marketing background. If I ever get caught up, I'll have more posts on this.

UPDATE: based on some feedback, I refined my position somewhat in the comments section.

Inequality

Frances Woolley has a post about inequality up in which she notes:

It's impossible for all firms to pay their CEOs above the median salary - by definition, half of executives must be paid below the median. If the majority of firms adopt a compensation policy like the Bell Canada Enterprises one quoted above, CEO salaries will increase inexorably. At the same time, allowing firms to bring in temporary workers at less than the prevailing market wage prevents the price of labour from being bid up in response to labour shortages, dampening salary growth for workers at the lower wage end of the labour market.
 

What I found interesting was not just the argument, but rather what happened in the comments.  People focused on the second piece of the argument (temporary workers at below mean wages) and whether the sense of justice should be local or global.  Consider Mike Moffat:

The inequality discussion changes a great deal if you consider the effects it has on Canadian inequality vs. global inequality. Why should the former necessarily be the lens we use to look at this problem? \
 

The question here is why is the focus on workers and not on CEOs?  I am a big fan of flexible immigration policies and I celebrate them.  But I wonder if a temporary workers program (at below market wages) isn't just a half-way measure.  Why not have permanent workers who have full rights to switch jobs? 

I am not sure that removing the best and brightest from the third world is always a good plan, but if we are going to do it then why not make it easy for them to stand as equals in the society they are helping to build? 

Why Observational Epidmeiology is frustrating

Andrew Gelman has a post up on the history of cigarette smoking research, based on a book he was reading a while back.  It's pretty interesting but what really caught my eye was this comment:

Vague statistical inference can not possibly establish such a causal link. Even valid associative inference should establish a 50-100% correlation between smoking and cancer, but it does not even come close. Most people who smoke don’t get lung cancer, and at least 10% of Americans who do get lung cancer- do not smoke. There are also huge international/ethnic variations among smokers and cancer rates. There is currently no proof whatsoever for the alleged smoking-cancer causal link. None. Smoking is a disgusting and silly habit. But all that one can now objectively say is that it is a risk factor for cancer and increases the incidence of lung cancer.
 

And this was not the only person in the comments who was casting doubt on this association.  As an epidemiologist, I want to scream.  If people will not believe this evidence then they really will not believe any level of evidence for observational epidemiology.  We have cohort studies going back 50 or more years (Richard Doll has one). Even better, the members of this cohort did not initially know that smoking was harmful (and I recall that the original hypothesis was automobile fumes and not smoking, although my memory may be failing me here).  So we don't even suspect a healthy abstainer effect. 

The requirement for a 50 to 100 correlation seems to ask for smoking to be directly causal of lung cancer instead of increasing the underlying risk of lung cancer.  Consider skiing and broken legs.  Not all broken legs are due to skiing and many people ski and do not break a leg.  But there is no question that skiing is a risk factor for broken legs.  Another good example is collapsed disks in the back.  If you are working with a veterans population, the first question you ask when you see a compression fracture in the spinal cord is "were you a paratrooper?".  Not all paratroopers have compression fractures and not all compression fractures are due to jumping out of airplanes, but it is a pretty direct link to increased risk. 

There is a libertarian line of defense here: people ski because they value the enjoyment of skiing more than the risk of a broken limb.  I am not always delighted by it, but it is at least an arguable position.  But directly denying the link between smoking and lung cancer seems to be setting a very aggressive standard of proof. 

Felix Salmon is back: California pension edition

Felix has a very nice post on the new retirement program that California is considering.  I think it is a very good idea and the state running it is a major plus.  It's astounding but true that the very best instruments that I can find in the private sector (currently thinking Vanguard here) seem to run neck and neck with large retirement plans in terms of both fees and returns.

I am a little less thrilled by investment guarantees, unless they are really well adjusted for inflation.  But, conditional on that, this would also smooth out returns in a very nice way.  Sure, it has the government investing in the markets.  But the only other alternative is paygo, which has come under a lot of question as to the willingness of taxpayers to honor previous promises.

I agree that this is an exciting development.

This will sound like a snarky question, but it's not

Via a long chain starting here, came across the following article on Romney's missionary service:

Republican presidential candidate Mitt Romney could be excused for having flashbacks to the 1960s when he went door to door in Berlin, New Hampshire, on Thursday.

The former Massachusetts governor worked in France as a Mormon missionary from 1966 to 1968, one of the church’s thousands of earnest young men (mostly) who knock on doors and proselytize. At that point Romney had plenty of doors slammed in his face, but on Thursday, not so much.

“This is a lot easier,” Romney quipped to Reuters. “People speak English. They wish you Merry Christmas. They don’t think you’re a salesman. People used to come to the door [in France] and wag their fingers: ‘No, I don’t want anything.’”

Many French people at the time were “not happy to see Americans, because we were in Vietnam at the time. That was tough,” he added.

I realize the war was unpopular in most of Europe, but I'm a bit surprised to hear it was unpopular in France. given their relationship with the country and the conflict. I was under the impression (backed up by a quick visit to Wikipedia) that the US was, in a sense, picking up where the French had left off. I even seem to recall Eisenhower saying that we should be careful about getting involved with Vietnam because it was a continuation of a pro-colonial war (am I remembering this right?).

How strong was the anti-war feeling in France in the late Sixties?

This is getting old,,

I've been getting this this screen frequently and seemingly randomly (it seems to have a special fondness for Economist's View). Is this a Chrome thing? A Microsoft thing? A techie conspiracy to get back at me for that whole ddulite business?

What's wrong with the blogosphere?

After the GOP convention, are they actually letting this pop culture allusion go by without pounding it into the ground?

Smart meters, incentives and a chance to use "extruded" in a post

Here's a product idea for you, an old notion repackaged for a new problem. It's a piece of plastic in the shape of an extruded upside-down T filled with a saline or alcohol solution that freezes exactly at a certain temperature (let's say 20 degrees). During the summer, you put the T into your freezer (I'm assuming the freezer is on the top) and set the thermostat timer (did I mention you need a thermostat timer?) to turn the freezer down to ten degrees (I'm using round numbers here. Actual values would certainly be different) from one a.m. to five a.m. then up to just under twenty degrees from five to ten then up to twenty-five from ten to four then just over twenty from four till one the next morning.

Obviously, I didn't put a lot of research into freezers before coming up with these temperatures, but I think you catch the basic idea: have the freezer work hardest in the middle of the night when electricity is cheap and the house is cool and do as little as possible when the grid is overloaded.

There's nothing original here. Old-fashioned ice-cream makers use the same principle. Nor is there anything unusual; without breaking a mental sweat, you could probably come up with a dozen comparable (or better) ideas for smoothing out the power demand curve.

By making more efficient use of our electrical resources, these ideas would pay for themselves in... well, never. You pay the same for the electricity you consume at two in the morning as you do for what you use just before a brown-out. That's the problem. There are lots of things we could do to smooth out our power demand curve but under the current system (pardon the pun) there's no incentive.

There's a notable precedent we can refer to. The current generation of of automobiles feature an array of innovative energy efficient technologies. The feature that impressed me most as a driver was the continuously variable transmission. An engineer pointed out that many of those energy saving technologies, including the CVT, had actually been around for decades, but had only come into wide use because regulations and/or high prices (think Europe) and consumer demand had forced engineers to find ways to squeeze every possible mile out of a gallon of gas. We can argue about what kind of incentive worked best to spur innovation, but no one can argue that incentives didn't work.

Of course, we do have the technology for variable demand-based pricing, and we have a line of incredibly cheap credit to pay for implementing it. What we don't have is a real interest in solving problems.

And a lack of interest in solutions is a difficult problem to solve.

is $152B < $186B

Opinions differ.

Urban Libertarians

This comment thread go me thinking about something that's been bouncing around my head for a while now. Living in a city or a suburb puts severe limitations on what it means to own a house. (the case is even more extreme with condos but I don't really consider that ownership. Those are just apartments with mortgages.) Between HOAs and city ordinances, you have remarkably little freedom to do what you want with your own property.

That's just the start. Life in the city is constrained by a dense array of rules, because

1. Even as a homeowner, you spend a great deal of time in property you don't own;

2.  Cities encourage specialization and a correspondingly high degree of interdependence;

3. It takes a lot of rules to allow this many people to live in this close a quarters;

4. If my freedom to move my arm stops at the closest person's nose, crowds have to severely limit my freedom.

I'm a city dweller, but if I gave into my libertarian tendencies, the only place I could stand living is back in the country. I'd get a place in the lower Ozarks off highway 7 at the end of a dirt road where a man can do what he damned well pleases. I'm not planning on moving any time soon but there are times the notion hits me (besides, it bothers me that I've forgotten that distinct smell of a dirt road).

Which is why libertarian urbanists like Edward Glaeser confuse me so. If the really value liberty so highly, why do they embrace the most rule dependent of lifestyles?