Sunday, March 15, 2015

I don't have time to discuss it today...

but you should definitely check out this post from Dean Dad if you've been following the higher ed side of the education reform debate.

Public Matters: A Response to Kevin Carey

Friday, March 13, 2015

More on the rise of PR

Living in North Hollywood, one gets plenty of notice of the upcoming Emmy season. Perhaps even more then with the Oscars, this award show is preceded by a blanketing of the neighborhood in "For your consideration" billboards. You can get a rough but reasonable idea of who is spending what by looking at how many billboards you see for different shows and different networks.

Per show, at least, there seems to be a huge disparity between Netflix originals and virtually everybody else. Billboards for House of Cards or Orange Is the New Black are all over the place. By contrast, I don't recall seeing an Emmy ad for Orphan Black or Justified or even the Good Wife. On this per show basis, it would certainly appear that the Emmy-season outdoor advertising budget for Netflix is many times larger than that of its nearest competitor.

I don't want to get into the question of whether or not this is a good business decision on the part of Netflix and I certainly don't want to open the topic of which awards were deserved. Instead, I want to tie this into the previous post on the rise of PR and the decline of journalism.

I read any number of pieces about how winning Emmys meant that Netflix had "arrived." As far as I can remember, none of these articles mentioned the disproportionate level of marketing it took to win these awards. Of course, omitting context is a common sin, particularly when the details undercut the standard narrative (adherence to the standard narrative is pretty much the prime directive of modern journalism), but there is an added layer of conflict of interest here.

The practice of letting interested parties research stories and even write copy is as old as typesetting, but there is reason to believe things have gotten much worse. What was once an occasional lapse now appears to be the norm.

Modern journalism is now basically row upon row of glass houses. Stone-throwers have become decidedly unpopular (check out the NYT's attitude toward Nate Silver). Even if a reporter wasn't beholden to some publicist, he or she would still face considerable pressure from colleagues and editors not to make a big deal of these questionable relationships.

I realize I seem to pick on Netflix a lot, but I really don't have a serious problem with the company.  My problem is with the way today's journalists cover business, neglecting due diligence, allowing conventional wisdom to outweigh facts, and letting companies write their own version of reality.

Netflix just happens to be a great example.

Thursday, March 12, 2015

A premature diagnosis of cost disease?

After cuts in state funding, the most popular theory to explain the rapid increase in college tuition seems to be cost disease:
Baumol's cost disease (also known as the Baumol Effect) is a phenomenon described by William J. Baumol and William G. Bowen in the 1960s.[1] It involves a rise of salaries in jobs that have experienced no increase of labor productivity in response to rising salaries in other jobs which did experience such labor productivity growth. This seemingly goes against the theory in classical economics that wages are closely tied to labor productivity changes.
I've never felt entirely comfortable with the way this explanation fits (or fails to fit) the data. It always seemed to me that the tremendous increase in very low-priced adjunct labor would more than balance out the flat productivity gains.

If this post by Paul Campos of Lawyers, Guns and Money is accurate, the theory is even more at variance with the facts that I thought. ( Campos also has some interesting things to say about the drop in state funding explanation.)
Everyone is aware that the cost of going to college has skyrocketed since [fill in any date going back to the middle of the last century]. Why has this happened? This post is about one possible explanation, that turns out not to have any validity at all: increases in faculty salaries. In fact, over the past 40+ years, average salaries for college and university faculty have dropped dramatically.

Salaries have increased, sometimes substantially, for a tiny favored slice of academia, made up of tenured professors at elite institutions, some professional school faculty (business, law, medicine), and most especially faculty who have moved into the higher echelons of university administration. Such examples merely emphasize the extent to which the economics of the New Gilded Age have infiltrated the academic world: the one percent are doing fabulously well, and the ten percenters are doing fine, while the wretched refuse of our teeming shores will adjunct for food.

Numbers:

Average salary for all full-time faculty in degree-granting post-secondary institutions (this category includes instructors and lecturers, as well as all ranks of professors) in constant 2012-13 dollars:

1970: $74,019

2012: $77,301

These figures, of course, give a very incomplete picture of the economic circumstances of the actual teaching faculty in America’s institutions of higher education.

One of the more astonishing statistics regarding the economics of our colleges and universities is that, despite the fantastic increase in the cost of attending them, there are now on a per-student basis far fewer full-time faculty employed by these institutions than was the case 40 years ago. Specifically, in 1970 nearly 80% of all faculty were full-time; by 2011, more part-time than full-time faculty were employed by American institutions of higher learning (note that the former category does not include graduate students who teach).

While comprehensive salary figures for part-time faculty aren’t available, it’s clear that their salaries are on average vastly lower than those of full-time faculty (and of course when it comes to who does the bulk of the actual teaching at many schools, the designations “full-time” and “part-time” have a distinctly Orwellian flavor). If we assume that “pat-time” faculty earn one-third as much as their full-time counterparts — and this seems improbably optimistic, given that the average compensation for part-time faculty for teaching a three-credit course is around $2,700 — that would mean that in 1970 average salaries for college and university faculty were nearly 30% higher, in real dollars, than they are today.

This an astonishing figure, given that, in the last 40 years, tuition at private colleges has more than tripled, while resident tuition at public institutions has nearly quadrupled.

You guys can write this post yourselves -- I'm tired

http://www.nytimes.com/2015/03/10/opinion/david-brooks-the-cost-of-relativism.html






Wednesday, March 11, 2015

More Martian musings – – reality shows and diet pills

Given all of the renewed attention to the Mars One project, this might be a good time for a quick little catch-up essay.

Maybe it's just me, but there are a few extremely salient points that have a way of being neglected in this conversation.

First, manned interplanetary spaceflight is almost certain to be very expensive and the cost for setting up permanent self-sustaining colonies is almost certain to be many times more so.

Second, the superiority of manned versus unmanned spaceflight is, for now, almost entirely symbolic. This does not mean that there are not certain specific economic and scientific benefits associated with manned spaceflight nor does it mean that manned spaceflight is a bad idea. It just means that, given current technology, sending explorers to Mars is something that, in the final analysis, we would do because we choose to as a society. This is even more true with sending colonists.

I actually don't have a problem with this kind of argument. At the risk of some muddleheaded nostalgia, I like the idea of leaders standing up and asking the people what kind of country we'd like to live it. Though I am not a huge fan of JFK, I greatly admire both the rhetoric and the sentiment behind "not because they are easy, but because they are hard."

Which brings me to my main objection to Mars one.

Without delving too deeply into the promise and the limitations of businesses like space X, when it comes to the kind of massive operations we're talking about here, we really only have two choices:

The first is to decide this is something we want to do and that we are willing to spend a considerable amount of money it would take to do it;

The second is to wait for a technological breakthrough which will change the underlying economics, with the understanding that this breakthrough may not come through in our lifetime.

I don't want to wait into that debate right now but, if landing on Mars is important, then it is a debate we need to have.

Though every major aspect of the Mars One proposal is laughably unrealistic, it resonates with people because it gives us an out. We can sit around and enjoy dreaming about how exciting the future will be without actually having to make any of the tough choices or do any of the hard work to make it exciting.

The idea of sending people to Mars just by watching a reality show is analogous to the idea that you can solve a lifelong problem with obesity by taking a miracle diet pill. I suspect that most of the people who try these products know on some level that it is foolish to trust the unlikely and unverified claims of late night TV pitchmen but the desire to believe outweighs their judgement.

Buying a proposal for a space program from a reality show producer isn't all that different.

Tuesday, March 10, 2015

The Decline of Journalism and The Rise of Public Relations

Any comments to a recent post, Andrew Gelman brought up a point that I want to dig into a bit more, at least briefly, the connection between the decline of journalism and the rise of public relations.

Here's my take: there is clearly a powerful relationship here though the direction of causality gets a bit complicated and goes both ways. For a variety of reasons, including but not limited to downsizing, an increasingly insular culture, and a shift to a star system that serves to hollow out the middle of the profession, journalism became both less diligent about maintaining quality and hungrier for free content (an appetite greatly expanded by online forums). While things changes were happening, companies were also growing more experienced at measuring and manipulating public opinion.

The decline in journalism created an extraordinary opportunity for corporate PR departments. News stories that portrayed products and companies in a favorable light were both more persuasive than traditional advertising and considerably cheaper.

While we are on the subject, my biggest concerns about the role of PR in modern journalism are not the question of accuracy or bias, though both of those are important. What really concerns me is the way these outside influences determine what does and does not get covered and the lack of awareness (or at least acknowledgement) on the part of the press. More on that coming later.

Monday, March 9, 2015

MOOCs and the Eugen Weber Paradox

As always seems to happen when I have other things I need to be doing, all sorts of interesting threads have started popping up and saying "Blog me! Blog me!"

Case in point, Erik Loomis of LGM has gotten back on the MOOC beat. I've got a couple of original posts on the subject in the works, but first I want to bring an old post from the teaching blog back into the conversation. It addresses what I think may be the fundamental questions of the ed-reform-through-technology debate:

After over a century of experimenting with educational technology, why have the results up until now been so underwhelming?;

And how will the new approaches being proposed fix the problems that plagued all of those previous attempts?


The Eugen Weber Paradox


If you follow education at all, you've probably heard about the rise of online courses and their potential for reinventing the way we teach. The idea is that we can make lectures from the best schools in the world available through YouTube or some similar platform. It's not a bad idea, but before we start talking about how much this can change the world, consider the following more-serious-than-it-sounds point.

Let's say, if we're going to do this, we do it right. Find an world renowned historian who's also a skilled and popular lecturer, shoot the series with decent production values (a couple of well-operated cameras, simple but professional pan and zoom), just polished enough not to distract from the content.

And if we're going to talk about democratizing education, let's not spend our time on some tiny niche course like "Building a Search Engine." Instead, let's do a general ed class with the widest possible audience.

If you'll hold that thought for a moment...

A few years ago, while channel surfing in the middle of the night, I came across what looked like Harvey Korman giving a history lesson. It turned out not to be Korman, but it was a history lesson, and an extraordinarily good one by a historian named Eugene Weber, described by the New York Times as "one of the world’s foremost interpreters of modern France." Weber was also a formidable teacher known for popular classes at UCLA.



The program I was watching was “The Western Tradition,” a fifty-two part video course originally produced for public television in 1989. If you wanted to find the ideal lecturer for a Western Civ class, it would probably be Eugen Weber. Like Polya, Weber combined intellectual standing of the first order with an exceptional gift and passion for teaching. On top of that, the Annenberg Foundation put together a full set of course materials to go with it This is about as good as video instruction gets.

All of which raises a troubling question. As far as I know, relatively few schools have set up a Western Civ course around "the Western Tradition." Given the high quality and low cost of such a course, why isn't it a standard option at more schools?

Here are a few possible explanations:

1. Medium is the message

There are certain effects that only work on stage, that fall strangely flat when there's not an audience physically present in the room. Maybe something similar holds with lectures -- something is inevitably lost when moved to another medium.

2. Lecturers already work for kind words and Pez

Why should administrators go to the trouble of developing new approaches when they can get adjuncts to work for virtually nothing?

3. It's that treadmill all over again

You probably know people who have pinned great hopes on home exercise machines, people who showed tremendous excitement about getting fit then lost all interest when they actually brought the Bowflex home and talking about exercise had to be replaced by doing it. Lots of technological solutions are like that. The anticipation is fun; the work required once you get it isn't.

This is not a new story. One of the original missions of educational TV back in the NET days was to provide actual classroom instruction, particularly for rural schools.* The selection was limited and it was undoubtedly a pain for administrators to match class schedules with broadcast schedules but the basic idea (and most of the accompanying rhetoric) was the same as many of the proposals we've been hearing recently.

Of course, educational television was just one of a century of new media and manipulatives that were supposed to revolutionize education. Film, radio, mechanical teaching machines, film strips and other mixed media, visual aides, television, videotape, distance learning, computer aided instruction, DVDs, the internet, tablet computing. All of these efforts had some good ideas behind and many actually did real good in the classroom, but none of them lived up to expectations.

Is this time different? Perhaps. It's possible that greatly expanded quantity and access may push us past some kind of a tipping point, but I'm doubtful. We still haven't thought through the deeper questions about what makes for effective instruction and why certain educational technologies tend to under-perform. Instead we get the standard ddulite boilerplate, made by advocates who are blissfully unaware of how familiar their claims are to anyone reasonably versed in the history of education.

* From Wikipedia
 The Arkansas Educational Television Commission was created in 1961, following a two-year legislative study to assess the state’s need for educational television. KETS channel 2 in Little Rock, the flagship station, signed on in 1966 as the nation's 124th educational television station. In the early years, KETS was associated with National Educational Television, the forerunner of the current PBS. The early days saw black-and-white broadcasting only, with color capabilities beginning in 1972. Limited hours of operation in the early years focused primarily on instructional programming for use in Arkansas classrooms


More on Mars One -- I expect this from ABC News but Sheldon?

There's been another wave of PR in support of the privately funded "Mars mission" Mars One (and yes, I do need to use quotation marks). There have been news stories, interviews with applicants who did or didn't qualify for the "mission," (NPR, how could you?) and even fictional characters like Castle and, sadly, Sheldon Cooper ("The Colonization Application").

Just to review, not only is this mission almost certain never to happen, but every major aspect of it collapses under scrutiny.

The funding goals are wildly unrealistic, the budget estimates are comically optimistic, and what little technology has actually been proposed is so badly designed that, according to an MIT study, it would be likely to kill all the colonists within a few months. I am pretty sure Howard would have pointed all of these things out to Sheldon.

You could also find some these objections in this piece from ABC, but you'd have to look closely because the reporters buried them as deep as possible, just far enough from the end to allow Mars One CEO/confidence man Bas Landorp have the last word.

Obviously this is a fun story, a lottery where anyone can become a colonist to Mars made even more dramatic by the twist of being a one-way trip. I also get that this is a story many probably most of us would like to believe. That is a high enough standard to justify a hook on a TV episode, but it is an embarrassingly low one for major news outlets.

Friday, March 6, 2015

The Golden Age of PR

I usually check out Jonathan Chait's blog once or twice a week and I usually ignore the "Most Viewed Stories" column to the right of the page. Recently, though, one of the items caught my eye.

"Who Was That at the End of the New Avengers Trailer, and Why Should You Be So Excited?"


The link led to a Vulture.com post about the comic book character the Vision, followed by some speculation about what part he might play in the upcoming Avengers movie. Looking over the article, it struck me that this is an amazing time to be a publicist.  We have gone from publications hyping movies to hyping trailers to hyping two-second shots in trailers.

Thursday, March 5, 2015

Cartoon Metalogic

Bud Grace, creator of the comic strip the Piranha Club, is a former nuclear physicist -- No, really, look it up -- and every now and then a bit of STEM humor pops up.

For investors, all money isn't created equal

In our most recent discussion of driverless cars, I made the following assertion about Google:
Google has a lot of reasons to want to be seen as a diversified, broadly innovative technology company, rather than as a very good one-trick pony cashing in on a monopoly (possibly two monopolies depending on how you want to count YouTube). A shiny reputation helps to keep stock prices high and regulators at bay.
I didn't really think of it at the time, but this concern is a point we have hit tangentially in the past and which probably deserves a bit more direct scrutiny. Investors often care a great deal about where a company's money comes from. This concern is often neither rational nor consistent and often leads companies to mislead the public about the makeup of their revenues.

Here are a couple of examples. I am going to be rather vague about some details because, you know, lawyers, but the broad outlines are both accurate and the circumstances are common enough that I could always find other cases if pressed . The first involved a financial services company that had products for customers at both ends of the economic spectrum. If you were to look at the company as an outsider or even as a new employee, you might very well assume that the two divisions were roughly equal. You might even suspect that the upscale was more profitable.

In reality a large majority of company's profits came from the low end. It turned out that the profit margin for providing services for poor people in this case was much higher. Stockholders, however, did not particularly like products that served this demographic. Also having a heavily promoted line of products for upper-class people did wonders for the stock price.

Here's another example:
The bank in question was in the middle of a very good run, making a flood of money from its credit card line, but investors kept complaining that the bank was making all that money the wrong way. This was the height of the Internet boom but the bank was booking all of these profitable accounts through old-fashioned direct mail. If it wanted to maximize its stock price, the bank needed to start booking accounts online.

The trouble was that (at least at the time) issuing credit cards over the Internet was a horrible idea. The problem was fraud. With direct mail, the marketer decides who to contact and has various ways to check that a customer's card is in fact going to that customer. With a website, it was the potential customers who initiated contact and a stunning number of those potential customers were identity thieves.

The Internet was an excellent tool for account management, but the big institutional investors were adamant; they wanted to see the bank booking accounts online. Faced with the choice between unhappy investors and a disastrous business move, the company came up with a truly ingenious solution: they added a feature that let people who received a pre-approved credit card card offer fill out the application online.

Just to be absolutely clear, this service was limited to people who had been solicited by the bank and based on the response rates, the people who went online were basically the same people who would have applied anyway. From a net acquisitions standpoint, it had little or no impact.

From an investor relations standpoint, however, it accomplished a great deal. Everyone who filled out one of those applications and was approved* was counted as an online acquisition. Suddenly the bank was using this metric to bill itself as one of the leading Internet providers. This satisfied the investors (who had no idea how cosmetic the change was) and allowed the bank to continue to follow its highly profitable business plan (which was actually a great deal more sophisticated than the marketing techniques of many highly-touted Internet companies).

*'pre-approved' actually means 'almost pre-approved.'
Put bluntly, companies will often pursue strategies or introduce products that are profit neutral or worse because these strategies and products make the companies look diversified or forward thinking or poised to take advantage of some major opportunity. Investors reward these perceptions. With this fact in mind, you can make sense of all sorts of strange business decisions.

For example, Amazon is an innovative, well-run. forward-thinking company, but its  P/E ratio (when it turns a profit) is often in the hundreds,* meaning the company has to be seen as being on the cusp of explosive growth. When you read about the company's online grocery service or its proposed drone-to-door deliveries** and you ask yourself how they can ever make a profit doing this on a large scale, the answer may be that they don't expect to.


* There may be some controversy over how  P/E ratio is calculated for Amazon but that's a topic for another post and probably another blogger.

** I added "drone-to-door" to emphasis the distinction between that proposed technology and large cargo drones. The latter actually does make business sense but would face huge regulatory hurdles.

Wednesday, March 4, 2015

Elegant theories and headless clowns -- more bad tech reporting from the New York Times

The previously mentioned Paul Krugman piece on opera singer Jenny Lind  included a link to this NYT article "adapted from The Price of Everything: Solving the Mystery of Why We Pay What We Do, by Eduardo Porter, an editorial writer for The New York Times." Krugman was criticizing the reliance on simple economic stories that don't fit the facts. Porter was telling one.
Baseball aficionados might conclude that all of this points to some pernicious new trend in the market for top players. But this is not specific to baseball, or even to sport. Consider the market for pop music. In 1982, the top 1 percent of pop stars, in terms of pay, raked in 26 percent of concert ticket revenue. In 2003, that top percentage of stars — names like Justin Timberlake, Christina Aguilera or 50 Cent — was taking 56 percent of the concert pie.
...

But broader forces are also at play. Nearly 30 years ago, Sherwin Rosen, an economist from the University of Chicago, proposed an elegant theory to explain the general pattern. In an article entitled “The Economics of Superstars,” he argued that technological changes would allow the best performers in a given field to serve a bigger market and thus reap a greater share of its revenue. But this would also reduce the spoils available to the less gifted in the business.

The reasoning fits smoothly into the income dynamics of the music industry, which has been shaken by many technological disruptions since the 1980s. First, MTV put music on television. Then Napster took it to the Internet. Apple allowed fans to buy single songs and take them with them. Each of these breakthroughs allowed the very top acts to reach a larger fan base, and thus command a larger audience and a bigger share of concert revenue.
Putting aside the fact that, as Krugman pointed out, we have examples of superstar musicians that predate both recording and broadcasting, this paragraph is still stunningly incomplete and comically ill-informed.

The 1980s cutoff is arbitrary and misleading. The 1880's would make more sense, though it really wasn't until the 1890s that things really took off and it has been a fairly steady stream of technological innovations since then.

Here's a brief, roughly chronological view of some of the highlights:

Disc records

Amplification

Radio

Optical sound tracks on film

Stereo

FM

LPs

HiFi

Television (which brought with it everything from Hit Parade to American Bandstand, Sullivan, Midnight Special and countless shows like this)

Cassettes

CDs

Recordable  CDs

Affordable digital audio editing

And then, of course, a whole family of internet-based innovations.

The past 125 years has been one long stream of "technological disruptions" for the music industry, but most of the innovation over the past couple of decades has mainly broadened the market by increasing selection and lowering production costs. In terms of "allow[ing] the very top acts to reach a larger fan base, and thus command a larger audience and a bigger share of concert revenue," at least for the North American and European audience, the top acts have been near saturation since the Sixties. (Check out the ratings for Elvis or the Beatles on Sullivan.)

By looking at the past thirty years of advances and ignoring the previous ninety, Porter gives us a blatant example of headless clown causal reasoning, arguing that x explains the difference in A and B because x is present in A while ignoring the fact that x is also present in B. Data journalism has fully embraced the idea that two numbers briefly moving in sync constitutes a causal argument.



The phrase "elegant theory" should have set off the red flags and warning lights. Elegance in these books pretty much always means "simplistic and unrealistic." The theories are aesthetically and emotionally appealing but they just barely fit the data in their examples and they usually fall apart completely when taken out on the road.

As previously mentioned, this goes back to what George Polya called (in a quote I really need to dig up) thinking like a mathematician . Polya suggested that the default setting of most people when presented with a rule is to look for examples while the default setting of mathematicians and scientists was to look for exceptions. Mathematical ideas get a tremendous amount of press these days but very few of the people covering them think like mathematicians.

Tuesday, March 3, 2015

Epidemiology Research

I am a big fan of Aaron Carroll, who often blogs at the incidental economist.  However, in his latest New York Times column he says:
Most of the evidence for that recommendation has come from epidemiologic studies, which can be flawed.

Use of these types of studies happens far more often than we would like, leading to dietary guidelines that may not be based on the best available evidence. But last week, the government started to address that problem, proposing new guidelines that in some cases are more in line with evidence from randomized controlled trials, a more rigorous form of scientific research.
So when have randomized controlled trials stopped being a part of epidemiology?  It comes as news to me, who has done this type of work as an epidemiologist.  In particular, there are threats to validity in trials as well, and a lot of smart causal inference research has looked at that as well.  Trials also have concerns about cross-over, attrition, and even valid design.  These elements are all part of a typical epidemiological education and are an important part of public health practice.  Even things like meta-analyses, where trials (and now sometimes observational studies) are pooled are typical parts of epidemiology. 

It seems like he wants to conflate that observational research = epidemiology. 

There is also a difference of estimands.  The trials can only assess interventions in diet and how they perform.  The real (true) intake of the participants is always approximated, except perhaps in a metabolic ward.  Even doubly labeled water studies need to make assumptions. 

The real bug-bear of nutritional research in humans is measurement error.  It is present in all studies (even trials, which are much less susceptible to bias than cohort or case-control studies).  That is a lot of what we struggle with in this research area. 

Now, it is true (and I agree with Aaron Carroll completely) that the trials tell us a lot of what we want to know.  In a real sense we want to know how dietary interventions, as they will actually work out in reality, will change outcomes.  So I share his concern that the trials seemed to be overlooked by people writing guidelines.  Or, in other words, I think his main conclusion is quite sensible.

But let's not forget the observational research is critical to understanding patterns of diet that create the hypotheses and interventions that can actually be tested in trials.  They also give a lot of understanding into how people consume food in the state of nature.  I am never going to stop focusing on high quality data and the most rigorous possible study designs.  But I think it would be wiser to represent the eco-system more completely. 

On the other hand, I am not an expert in health care communications, and it might be that such broad strokes are necessary when focusing on the general public.  After all, improving public health is everyone's goal, and I am happy to take a few "hits" if that is the ultimate outcome.  But I think it's also a challenge to think about how to make this type of research, and the nuances in it, better understood in general.

I have a lot of thinking to do. 

Monday, March 2, 2015

Defining away concerns about charter school attrition

[New information has come in and we may be making some changes to this post.]

After what seems like a long time, we are back on the bad education statistics beat. Joseph kicked things off with this post discussing some recent charter school research, particularly this paper by Angrist et al. I followed by reposting a couple of earlier pieces on attrition.

If you didn't see them when they came out, I strongly recommend you take a minute a read those two reposts (Selection effects on steroids and Selection on Spinach*). This is a big, incredibly complex story and it makes much more sense if you come in with some context.

I also want to say explicitly that I am not singling out the Angrist paper for criticism. It is, if anything, above average for the field; that's the scary part. I have a number of concerns about this study but they are all problems that you find in much, if not most of the research on charter schools.

Let's start with attrition and this passage from the paper. The first half of the paragraph mostly seems to be pretty good, except for one red flag [emphasis added].
A second potential threat to the validity of lottery-based estimates is the differential loss to follow-up between winners and losers (also called differential attrition). Students in our study are lost to follow-up if they are missing the MCAS score data we use to measure charter achievement effects. This usually happens when a student moves out of state or to a private school. Attrition can bias lottery-based estimates if different types of students are more likely to leave the sample depending on lottery results.
There are a couple of fairly subtle points here (since I'm not an expert on this research you might want to dig up a copy of the original paper -- I believe mine is behind a firewall -- and check my work). The first centers around the various reasons why a student might miss one or more standardized tests. The researchers do deserve some credit for mentioning the private school option but the don't seem to quantify it, nor do they mention reasons like changing schools which are much more likely than interstate moves to interact in a problematic way.

Easier to miss but far more important is the defining of attrition as leaving the data set rather than leaving the program. This isn't necessarily wrong but it's incomplete and worrisome in at least two ways: first because it differs from what we might call the standard definition. If you Google "charter school student attrition," you will generally find stories about students leaving charter schools and moving to other schools; second because the more common definition of attrition is far more likely to cause problems that can invalidate this study.

The rest of the paragraph is more troubling.
 For instance, losers might be more likely to leave than winners, and highly motivated students might be more likely to opt for private school if they lose. We therefore compare the likelihood that winners and losers have an outcome test score in our data. There are no statistically significant differences in follow-up rates in the lottery sample schools, a result documented in Appendix Table A3. It therefore seems unlikely that differential attrition has an impact on the lottery-based results.
That "seems unlikely" is very hard to justify. Putting aside for a moment, the issue of definitions, you can't control for differential attrition this way. It is entirely possible for two groups to have roughly the same level of attrition and yet have the selection effects going in opposite directions. Furthermore, the kind of highly selective attrition we're talking about here is very powerful (particularly if you throw in peer effects). Even if the selective attrition is limited to one group, it is entirely possible for a statistically insignificant difference in attrition rates to led to a substantial difference in outcomes.

(Perhaps it is just a coincidence but it seems that, as economists have played more and more the role of statisticians-at-large, we seem to be seeing more of these "don't worry, everything will balance out" assumptions.)

I want to be careful with the next part because as mentioned before, I'm not an expert in this field nor have I gone through the paper in great detail, but think about the following line from the paper:

"The effects of charter attendance are modeled as a function of years spent attending a charter school."

Keep in mind that we appear to have a lot of cases of charters (particularly those with the 'no-excuse' model) singling out out students who are likely to do poorly and either forcing them out of the program or encouraging them to leave voluntarily. This probably means that a lot of students who would have been low-score/high-charter-years had they stayed where they were assigned by the lottery have been shifted to the the low-score/low-charter-year category.

This isn't my biggest concern with this research -- it isn't even my second biggest -- but it is enough to potentially undermine the validity of the research.


Sunday, March 1, 2015

Nimoy tribute on MeTV

One of these days, I would love to spend some time discussing the many clever ideas of Weigel Broadcasting. (Keep in mind, Carl Reiner has called MeTV's promos "brilliant.") The company provides a fascinating case study of a well-run business.

Unfortunately, this post is time-sensitive, so I'll limit myself to a quick DVR alert.



I particularly recommend the Man From UNCLE episode, which also features William Shatner and Werner Klemperer and is simply a lot of fun.

MeTV also ran the Star Trek episode "Amok Time" last night and dusted off this irreverent but affectionate spot.





It's the kick that sells it.