Value added testing

This article is worth reading throughout.  However, the most relevant passage is:

Because student performance on the state ELA and math tests is used to calculate scores on the Teacher Data Reports, the tests are high-stakes for teachers; and because New York City uses a similar statistical strategy to rank schools, they are high-stakes for schools as well. But the tests are not high-stakes for the eighth-graders at Anderson. By the time they take the eighth-grade tests in the spring of the year, they already know which high school they will be attending, and their scores on the test have no consequences.

Seriously, can no student of incentives not see how this could go terribly, terribly wrong?  The students could, for example, decide to blow the test because the teacher was overly rigorous. 

This is why I am less skeptical about metrics like the SAT.  It is still flawed as all observational research is tricky to derived unbalanced estimates from.  But in this case, the students, the teacher and the school all have something at stake in their performance on the exam. 

They're just not paying Gelman's preferred currency -- the odd economics of access

This post from Andrew Gelman on a truly bad graph and this story from NPR had a common element that caught my eye, partly because I've been thinking about the role connections play in careers and economic mobility.

Here's the relevant passage from Gelman:

All ugliness aside, this reminds me of a story that I haven’t had a chance to share . . . until now. The bit on the above infographic about “C-level execs” reminds me of something that happened a couple years ago, when I was invited to speak at an event, “called “The 2012 Election: Predicted Outcomes and Implications for Your Business,” for “an association of C-suite executives.” I responded that I was interested and told them my fee, to which they responded:

"I am glad to hear that you are interested in this opportunity. However, we don’t compensate speakers as we find that most are interested in the opportunity to be in front of a room full of high quality C-suite executives."

Is that for real? I speak to all sorts of people for free (for example, I just spoke last month at a datatviz meetup), but I was surprised to hear that an association of business executives weren’t planning to pay. So I declined. I say this not to imply that I’m some sort of anti-corporate crusader (after all, I would’ve been happy to talk for pay!) but to express my bafflement at the whole “C-level executive” thing. What’s going on with that?

And here's the opening of the NPR story:

CNBC is far and away the television ratings leader in the financial cable news business. Now, evidence arrives that its executives, producers and reporters are going to great lengths to maintain its status.

The channel has adopted a policy that prohibits guests from appearing on rival channels amid breaking news if they want to be seen by CNBC's larger audience.

In both cases, it seems that the party providing the service is, in a sense, paying the party receiving it. The Chief ____ Officers would receive actionable business intelligence; CNBC would receive valuable on-air talent. You would expect the recipients to try to compensate for these services either directly on indirectly. Instead the company in the first anecdote insists on getting for free what would normally be a well compensated service while CNBC actually puts additional demands and restrictions on the people providing the free service.

Of course, that's not what's going on, at least not most of the time. While Gelman really was a researcher who was offering to present findings of potential value to a company, most similar presentations are actually sales pitches. I'm going to be a bit vague here because I'm currently working in the corporate world, but without naming names, tremendous amounts of capital is drained from American businesses every year by researchers with a slick HBR paper, an inspirational story and a new set of buzz-words.

If you can get a C-level executive interested in your product you can charge obscene amounts of money for years. Better yet, the metrics for success are ill-defined to nonexistent. The only real barrier to becoming the next Tom Peters is access to someone at or near the top of a company. It's easy to understand why companies think they can get research presented for free.*

Something similar is going on in the CNBC example. The majority of the people interviewed on CNBC are either selling something -- a book, some stocks, themselves -- or are building and maintaining a brand. Doing these things requires getting access to large numbers of people. You could manage this by buying advertising (most effectively on a targeted platform like CNBC) but ads are expensive and, even before TiVo they were often avoided either through channel surfing or, before the remote, simply leaving the room for a snack or a bathroom break.

Getting your access as part of regular programming and having it labeled as news is a far better way of doing things. You can why the interviewees would put up with restrictions.

This does, of course, raise questions about conflict of interest and journalistic integrity, but that's a topic for another post.



* This still leaves unexplained why executives would want to, as Gelman points out in this comment, waste their time with inferior speakers to avoid paying a relatively trivial amount of money. Thoughts on that will have to wait for an upcoming post.

UPDATE: This comment from Felix Salmon throws some amusing light on an event similar to the one in Gelman's anecdote.

Education Research

A strong opinion:

I don’t have an informed opinion on the standard of observational research in education, but if the standard is high, there can’t still be lots of low-hanging fruit in the form of cost-neutral interventions whose benefit is obvious without comparative evaluation.

I think that this position is on the strong side given the long lags between exposure (i.e. education) and outcomes (i.e. adult competencies).  But it does point out that easy and cheap interventions that are massively easy to measure are unlikely if there has been any sort of careful research.  So perhaps the randomization idea isn't all bad?

"More colleges stop giving credit for AP exams"

I have to admit mixed feelings about this (from Marketplace):

Yet despite all the hard work, students like Brown may not be able to place out of required college courses or even skip freshman year if they score well on the AP tests. Some prestigious colleges have stopped giving academic credit for AP tests scores.

Brown doesn’t. Columbia doesn’t, and most recently, Dartmouth said it won’t let AP students skip ahead.

“We want a Dartmouth education to take place at Dartmouth,” says school spokesman Justin Anderson.
...

Conley says prestigious schools can afford to be picky about what credits to accept. But there are “more general admissions schools where they want students to bring AP credits and they do want to reward them for doing that.” In other words, AP credit is like bait for the best students.

For various reason, I've always preferred CLEP to AP as a method of testing out and as for advanced classes, I tend to favor plans that actually allow students to attend college classes while in high school, but of the choices we normally see, AP is certainly better than many and I very much support the idea of testing out of lower level courses.

I'm sure there are some exceptionally good freshman level courses at Brown, Columbia and company but that section of Cal I would have to be pretty impressive to justify the expense, the time and the opportunity costs (I sure there's an even better course the student could use to fill that time).

"Cosma Shalizi - Why Economics Needs Data Mining"

As part of a comment thread to this post on big data, Mark Thoma sends us here:



UPDATE:

I should have mentioned this before, but, while the entire segment is worthwhile, I was particularly glad to hear the interviewer address the strange discrepancy between the way economists use "data mining"  and the way statisticians, computer scientists and most people in business use term.

Interesting stats blog

From New Zealand. 

Education blogging of note

David Warsh has posted another entry in a thoughtful series about early childhood development. He also includes useful links including one to this interesting interview with James Heckman,

Yeah, I'm getting tired of these too

Once again I made the mistake of listening to On the Media. It started out reasonably benign with a standard but harmless piece on cyber-security, then went downhill with a smug piece on fact-checking before going off a cliff with this ddulite puff piece. Here's the blurb:

Late last month, The Washington Post debuted "The Truthteller," an application that it hopes will soon be able to fact-check politicians' speeches in real time using speech-to-text technology and a vast database of facts. Brooke talks to Cory Haik, The Washington Post's executive producer for digital news, about the app.

Of course, the speech-to-text and database problems are trivial next to the issues with processing natural language. To work at anywhere near the level discussed by Haik, the system would have to be considerably more advanced than IBM's Watson. Watson was designed to address short, free-standing questions following similar linguistic conventions and having clear, unambiguous answers.

This isn't meant to denigrate the team that developed Watson. Just the opposite. Interpreting natural language is extraordinarily difficult and solving even highly constrained problems is an impressive and important accomplishment. IBM has a lot to brag about.

The Washington Post currently has a beta up that apparently can sometimes spot strings that look like simple factual statements that lend themselves to automated comparison to a database. There's no reason to believe the app will ever move much beyond that, but the interviewer did believe...

without sign of suspicion...

immediately after a segment boasting about how carefully On the Media checks its facts.

To quote Snoopy, "the mind reels with sarcastic replies."

When does original programming make business sense? part 1 -- cable

The big PR push (and credulous coverage) of Netflix's House of Cards has got me thinking about the necessary conditions for successfully launching an original scripted series. Before we get to the business case for Netflix, here are some thoughts on the current cable landscape.

When it comes to producing original content, the big players tend to fall into three groups (with one notable exceptions):

HBO
Showtime
USA
FX
TNT
Nick
Cartoon
AMC

Let's start slicing. Nick and Cartoon are certainly successful (SpongeBob is, by some standards, the most popular cable show ever) but animation and children's programming play by a very different set of rules. Production costs are extremely low and high audience turnover means that you can keep running the same few episodes for years. Furthermore, since children's programming works best in dedicated blocks, it's difficult for general interest channels to fully capitalize on this hugely profitable market. (Might throw Disney in here as well.)

Then there are the pay channels. Once again a very special business model. From a revenue standpoint, someone who watches five hours a month of Showtime counts the same as someone who watches fifty. What matters is loyalty. If you can bring in people with movies and specials, then get them sufficiently caught up in, say, Dexter, you don't have to worry about Starz stealing them away with better movies and specials. Better yet, if you can get fans of an older show like Dexter also hooked on a new show like Homeland, you can keep those monthly checks coming in for a long time.

This model does not work if your revenue comes from advertising. Having one show that a viewer never misses isn't nearly as valuable as having a bunch of shows that a viewer watches occasionally. This makes it extremely difficult for an expensive original cable show to pay for itself (particularly with a thirteen episode season). In this environment, viable shows have to add synergistic value. This brings us to the USA, FX and TNT, big flagship channels of ginormous media companies. These channels have the deep pockets, extensive synergies and long time frames that make it easier to make the business case for a Burn Notice or a Justified.

Which leaves AMC as the exception, the channel that would seem to suggest that anybody with good enough shows can make a go of original programming, but before an executive at AE points at AMC when greenlighting their next original drama, there are a couple of cautionary notes to keep in mind:

1. It's dangerous using outliers as examples

AMC got extraordinarily lucky twice, first in finding executives who could spot good shows and second in having good shows cross their desks. This is not something you can plan on in advance. In fact, it's something most companies fail at when they try.

2. (and this is the big one) It's not entirely clear that AMC is actually making money on this

It would take a good financial analyst and a lot of access to answer this definitively, but certainly looks like the network loses serious money on Mad Men and has been forced to make big and risky budget cuts to the show that is making them money, the ratings hit Walking Dead. Keep in mind, AMC has been both good and lucky. The odds against duplicating its success are slight and that success may not be all that, you know, successful.

These days, every cable channel seems to have an original scripted series in the works. From the executives' standpoint, this is easy to understand; these shows are the best way for a network to get prestige and media attention and who wouldn't want to be the VP who greenlit the next Breaking Bad? From the standpoint of the investor, though, if the channel isn't a TNT or an HBO or a Cartoon Channel, you might prefer another strategy.

Weekend cinema buff alert

Hulu has temporarily lifted the paywall around the Criterion Collection. If you're a movie lover, you should definitely drop by.

p.s. but be prepared for LOTS of commercials.

The best tech writing you'll read this week just might be an article about wheelbarrows

I may may not always show it but I'm a great fan of technology (it's tough love, but love nonetheless). What I am not a fan of is the way that we discuss technology. Many (maybe most) journalists on this beat are so besotted with the gee-whiz aspects that they have little time to think seriously about why are certain technologies successful, what demands they makes on infrastructure, and how they answer existing needs and create new ones. (Search on 'ddulites' for more on this).

Having journalists report on the subject of their infatuation is generally a bad idea, but in this case the damage goes beyond the inevitable annoying prose. Technology is important and when we can't discuss it intelligently we make bad decisions that end up holding progress back.

Look at Groupon. One of the most exciting and promising areas of research today is the study of social networks, but partly because business reporters did such a poor job covering the subject, this crappy gift-card company was able to convince investors that its creaky business model was 'social.' (and no, requiring minimum orders for a deal to go through wouldn't have qualified even if they hadn't set the threshold meaninglessly low). The money that went to Groupon was diverted from other investments, including businesses that actually used the properties and analyses of social networks in their business plan.

Perhaps it's not surprising that the best thought-out, most clear-eyed popular writing on technology I've seen recently would focus on the least glamorous type of technology imaginable (via DeLong).

More fun with charts

Daniel Kuehn and Joseph previously discussed this post by Megan McArdle entitled "Department of Awful Statistics: Income Inequality Edition." They both make good points, but I'd like to approach this from the angle of appropriate visualization. McArdle supports her thesis that the middle class is neither disappearing nor getting poorer with charts derived from census table H-17 which you can and really should download here (the best way to keep us all honest is to play along at home).

The trouble is they're bad graphs.

To the extent that statistics includes data visualization, this is definitely bad statistics. When trying to depict trends and relationships, you generally want to get as much of the pertinent information as possible into the same graph. You don't want to force the reader to jump around the page trying to estimate slopes and compare magnitudes, nor do you want to take a few snapshots when you can easily picture all the data.

There are lots of acceptable ways of laying out the data table H-17, but I'm going just going to go with the simplest (partly because I like simple and partly because I'm doing this on Openoffice). As with McArdle's graphs, the numbers are inflation-adjusted.

I'm not that comfortable with this data (for reasons I'll get to in a minute), but this does look fairly consistent with the hollowing out of the middle class with 35K-75K (the top two lines) dropping more or less steadily for decades. Also check out the more than fourfold increase of people making more than 150K,

The two main things that make me uncomfortable with the data are the start point (with falls close to at least a couple of inflection points) and, on a related note, the failure to account for the baby boom which was at the bottom of its earning power forty years ago and should be close to the maximum now.

As far as I can tell income distribution is not broken down by age in these tables (though I suspect the data are available on request). We can, however, answer the related question of what median income looks like when we control for age and extended over a longer interval. (Download table P8 from here)

You can see why I was nervous about starting in 1967.

The question of income inequality and what's happening to the middle class is a complicated one and is probably best addressed by people who know what they're talking about, but if you are going to try to argue one side of the case graphically, you should at least take the time to use appropriate graphs.

p.s. I picked 35-44 because it seemed like a good representative mid-career interval and because, since I wasn't comparing different age groups, an uncluttered one-line graph seemed sufficient. If you prefer, here's the multi-range version (though I don't know if it adds much information).

Thinking about failure and collective amnesia.

No. Not in the sad my-life-adds-up-to-nothing way, but more in the case study sense. I've been noticing how often optimistic analyses of proposed products and business models echo the same arguments used over the years for various underperforming enterprises and catastrophic failures, invariably without a flicker of recognition.

No doubt, this is partly due to a general lack of long-term memory in the pundit class, but the problem seem particularly acute when it comes to failure. There are exceptions like this well-thought-out analogy by Josh Marshall or this piece of historical context for Zucker's Leno debacle from Kliph Nesteroff, but as a rule, most journalists don't pay nearly enough attention to these counterexamples (which makes it all the more difficult to avoid repeating mistakes).

I'll try to add some more entries and drill down into some of the specifics, but in the meantime, here's a short list of some potentially useful examples ideas that seemed (and in some cases, actually were) good ideas at the time.

The aforementioned mentioned attempt to make Jerry Lewis king of the talk shows.

A late Eighties format that doubled the resolution of video tapes while being completely compatible with standard VHS.

An attempt to break the Seventies DC/Marvel duopoly.

An attempt to break the Coke/Pepsi duopoly (perhaps breaking duopolies deserves a subcategory).

Adios, Amiga.

Fun with charts

This post by Daniel Kuehn is worth reading, although all of the action is in the comments. 

I think he is right on about the denomintor problem in interpreting her graphs.  It's also a very good example of when a point can be correct and yet not explain all of the differences (her comments about rounding).  However, the labeled buldge seems to be a lesser sin than variable bracket sizes on a density plot.

As for the change argument, it is fine to use a chart to explain something and then talk about the expected changes to the distribution.  Where I am less happy is that there are changes going on in the United States all of the time (aging of the population, propensity to form a new household) that are going to influence the shape of this curve.  It is possible to imagine the curve shifting exactly as Jon Evans suggested, and the reasons having to do with factors that have nothigng to do with inequality.

But standardized curves have their own issues . . .  So even Magan's use of the curve to show the shifts over time doesn't address the null conditional on the changes in the underlying population.  This may even help her argument, I am not sure, but certainly I would rather graph density plots in equal sized segments just for reader clarity

Still, a worthwhile argument to follow and it is useful insofar as it improves understanding of what the plots do and do not mean. 

Annals of bad analogies

From here:

Think about it this way. Say your elderly mother had to be hospitalized for life-threatening cancer. The best doctor in the region is at Sacred Heart, a Catholic, private hospital. Could you ever imagine saying this? “Well, I don’t think our taxpayer dollars should subsidize this private institution that has religious roots, so we’re going to take her to County General, where she’ll get inferior care. ’Cause that’s just the right thing to do!”
No. You’d want to make sure that your tax dollars got your mom the best care. Period. Our approach should be no different for our children. Their lives are at stake when we’re talking about the quality of education they are receiving. The quality of care standard should certainly be no lower.

An analogy is the weakest form of argument, because it presumes similarities between cases.  In this case we are equating a one time event (cancer treatment) with a long term process (educating people).  There is also a difference in that cancer outcomes are much easier to measure (due to the fast time between diagnosis and resolution) than an educational process.  So "better" is much easier to evaluate.  Finally, it ignores magnitudes.  What is "better" and by how much.  Is it a matter of preference (Starbucks coffee is better than McDonald's coffee) or an objective metric? 


But this whole thing dodges the main question-- why is the County General hospital not competitive with the Sacred Heart hospital?  Is that not the more interesting question?  Is it because the County General can't turn patients away and so gets the sickest of the sick? 

These points matter.