Brief quote from EPI

I have a feeling I'm going to be referring to this a lot:

For a variety of reasons, analyses of VAM [Value Added Modeling] results have led researchers to doubt whether the methodology can accurately identify more and less effective teachers. VAM estimates have proven to be unstable across statistical models, years, and classes that teachers teach. One study found that across five large urban districts, among teachers who were ranked in the top 20% of effectiveness in the first year, fewer than a third were in that top group the next year, and another third moved all the way down to the bottom 40%. Another found that teachers’ effectiveness ratings in one year could only predict from 4% to 16% of the variation in such ratings in the following year. Thus, a teacher who appears to be very ineffective in one year might have a dramatically different result the following year. The same dramatic fluctuations were found for teachers ranked at the bottom in the first year of analysis. This runs counter to most people’s notions that the true quality of a teacher is likely to change very little over time and raises questions about whether what is measured is largely a “teacher effect” or the effect of a wide variety of other factors.

A study designed to test this question used VAM methods to assign effects to teachers after controlling for other factors, but applied the model backwards to see if credible results were obtained. Surprisingly, it found that students’ fifth grade teachers were good predictors of their fourth grade test scores. Inasmuch as a student’s later fifth grade teacher cannot possibly have influenced that student’s fourth grade performance, this curious result can only mean that VAM results are based on factors other than teachers’ actual effectiveness.

An experiment in blogging -- the conclusion

When assessing a statement, sometimes it's useful to rephrase it in a more general way and see how well it holds up. I tried that with a passage I found in a popular blog (one of the very few I read every day). Where the author had referred to members of a specific profession I substituted in the word 'employees' except when talking about unions ('employees unions' seemed redundant). I also changed a couple of words for consistency, but other than that the passage was exactly the same.

The resulting paragraph (seen below) was much more extreme than I had expected and it got me to thinking, how would people react to this passage if they encountered it without all the baggage? I decided to post the generalized version with a brief explanatory note then give people a couple of days to think about it before filing in the details.

Here's the generalized passage:

If you concede that employers need to be able to fire bad employees, then you can't fully defend the role of the unions. You can defend the concept of unions, and you can believe that some of the things unions do, like bargain for higher aggregate wages, help society. But most unions demonstrably make it very difficult to fire bad employees. That is currently a core function of unions, and something that must change. You're also going to need higher salaries to attract a better caliber employee into the workforce, and that's something unions could potentially help. But being "treated like professionals" has to mean both the opportunity to earn a good living if you do well and the potential to be fired if you fail.

And here is the passage Jonathan Chait (that's right, Jonathan Chait) originally posted in his blog:

If you concede that principals need to be able to fire bad teachers, then you can't fully defend the role of the unions. You can defend the concept of unions, and you can believe that some of the things unions do, like bargain for higher aggregate wages, help education. But most teachers unions demonstrably make it very difficult to fire bad teachers. That is currently a core function of teachers unions, and something that must change. You're also going to need higher salaries to attract a better caliber teacher into the profession, and that's something unions could potentially help. But being "treated like professionals" has to mean both the opportunity to earn a good living if you do well and the potential to be fired if you fail.

There are obviously two possible responses Chait could make here (three if you count ignoring it entirely). He could say he agrees with the general statement or he could argue that teachers are a special case and should be granted less union protection than, say, policemen.*

Ironically, the more defensible position Chait can take here is the extreme one, namely that unions should not do anything to discourage employers from firing their members. It's not a position that most readers of the New Republic would embrace but, as a statement of personal belief, it is extraordinarily difficult to rebut.

If he tries to explain why teachers constitute a special case, he will have to deal with the data and in this particular debate, the numbers are not his friends. (It's worth remembering that Diane Ravitch started out on Chait's side. Her road to Damascus came when she realized she could no longer reconcile those views with what she was seeing in the research findings.)

Jonathan Chait can be a formidable debater but he has shown himself to be largely ignorant of the research behind these issues (no one at TNR even knew enough about PISA to catch the bait and switch in the intro to Waiting for Superman and in the education debate that's about as slow as the pitches get).

He'll be trying to punch holes in the findings of institutions like EPI and Rand and big guns in the field like Donald Rubin. He'll have to show precipitous educational decline without resorting to the aforementioned PISA (good test but absolutely meaningless in this context). He'll have to explain why schools that use his policies are more likely to underperform than to outperform unionized schools. He'll have to justify firing people based on metrics so volatile that a third of teachers in the top 20% could find themselves in firing range the next year, metrics based on data so confounded that "students’ fifth grade teachers were good predictors of their fourth grade test scores."

This is one time the smart money is on the other guys.



* Yes, we fire policemen. What we don't do is is fire policemen based on unreliable metrics that are largely outside of the officers' control and are easily manipulated by their superiors

Cost of Higher Education

I thought that I would put some number onto Mark's previous post. According to this website, the median debt for a four year college degree is not small"

The median cumulative debt among graduating Bachelor's degree recipients at 4-year undergraduate schools was $19,999 in 2007-08.

Graduate school was more expensive and it's worth looking at the table to see some of the ranges. But the main point is clear: education isn't inexpensive for the student.

What does this mean if we want skilled workers? Well, to some extent the expected value of salary increases (adjusted for unemployment rate) has to be worth both the debt and the opportunity cost. It is fine to talk about education as "self-improvement" but not if it is a requirement for a position.

So I think Mark is correct that performance based bonuses would need to be fairly high to compensate for the additional risk a pure merit pay based system would place on new graduates.

Compensation and Career Paths

The economics of labor is complicated (yeah, I know... "And water is moist," but bear with me here). When you try to approach labor the way you would wheat (to use Shiller's recent example), you run into all sorts of problems. Wheat doesn't get its hopes up. Wheat doesn't get discouraged and pull itself off of the commodities market. And most importantly for this discussion, wheat doesn't make long term career plans.

With jobs requiring extensive qualifications, training and commitment, people tend to think in terms of careers and career paths. Career choices come with significant up front and opportunity costs (just ask someone who has made a major vocational change). In order to justify these costs, people generally look for careers that offer at least one of the following:

1. Opportunity for advancement;

2. Significant seniority-based raises;

3. Potential for really big payouts.

You might be able come up with some exceptions (priests, perhaps) but it's unusual to see people signing on to four or more years of college with the expectation that they'll be doing the same job for roughly the same pay from the year they graduate to the year they retire.

There's a caveat associated with seniority-based raises. They have to come with strong assurances of job security. Being both expensive and easy to fire leads to some fairly obvious problems.

It has gotten popular to suggest that we can solve any employment problem just by offering big enough performance bonuses, but if you're going to ask people to go into professions that offer stagnant pay, no security and no real chance for advancement, you'd better plan to put lots of zeros on those checks.

Contracts and compensation

Another relevant point on the economics of labor. This time from Felix Salmon (via guess who):

If I were Levin, I’d want a three-year contract commensurate with the sale price of Dealbreaker. Say a $100,000 signing bonus, and then a salary of $200,000 a year for three years. Contracts by their nature have to be lucrative things, because they carry an opportunity cost: if the Daily Show, say, came calling offering a television-size salary, Levin would have to say no if she was already under contract.

“Do your job. Keep employed. Don’t come up with a new idea.”

While we're on the subject of labor, take a look at this article from Robert J. Shiller (via Thoma, of course). It provides excellent background for the discussion and it's a fascinating read in its own right.

UNEMPLOYMENT, in this context, is like battlefield triage, leaving some severely injured soldiers to die so that medics can keep as many as possible in fighting condition. But, of course, such a harsh practice may not contribute to the best morale among those chosen to survive.

Unfortunately, managers often lay off more people than necessary, to ensure that they don’t have to repeat the ordeal anytime soon. The remaining workers must work harder, taking on some of the work of their missing colleagues, and productivity rises. (The economy today shows both increasing productivity and increasing corporate profits.)

Those relegated to unemployment can’t directly “poison the atmosphere” in their former workplaces. But they remain friends and neighbors of the employed, and their anger and distress, repeated in thousands of communities, contribute to a poisoning of the atmosphere of the entire nation.

Moreover, managers interviewed by Professor Bewley in the 1990s said that employees who hold onto jobs often suffer “survivors’ guilt.” They are genuinely pained, experiencing empathy with the less fortunate. In this troubled state, they don’t think about taking extravagant vacations, or buying new houses or fancy new cars. And this frugality detracts from demand that might produce jobs for others.

Similar thinking underlies the relatively low level of business expenditures today on buildings, equipment and software. Lower-level managers won’t ask for scarce resources for such things, because those items look like luxuries to fellow employees, who worry that there won’t be enough in the company budget for them to keep their jobs.

One top manager told Professor Bewley that he had to compensate for the reticence of lower-level managers, who won’t ask for anything. “I tell them to put in a few dreams for equipment they would like, because if they don’t try, they’ll never get what they want,” this manager said.

Of course, while that reticence may preserve jobs in one’s own company, it works against job growth elsewhere. A result is a loss of vigor in the aggregate economy, and the sapping of the very kind of creativity that might spur a recovery.

Professor Bewley shared with me a passage from an interview in July with a manager of a large manufacturing company. “There is more uncertainty, and everybody is afraid,” this manager told him. “Do your job. Keep employed. Don’t come up with a new idea.” In his own company, the manager said, “Everybody is doing the same thing.”

The cusp of coolness

One of the most popular genres of science writing since at least the age of Edison has been the "cusp of coolness" story, where the writer breathlessly tells us how some futuristic development is about to revolutionize our lives.

Here's the latest entry:

Although it may sound more sci-fi than sci-fact, a commercially developed jetpack is actually being eyed for mass production, with plans to eventually release it to the public. Let that sink in for a second. Jetpacks are real, and you might be able to buy one someday soon. Or at least see them among the skies.

I don't think we'll need the full second since jet packs have been around for between fifty and seventy years and you've been able to buy them for much of that time. The Germans had a prototype in WWII (Not surprisingly, Wikipedia has an excellent write-up on the subject). By the mid-Sixties they were flying over the World's Fair and showing up in Bond movies (yes, that was an actual Bell Rocket Belt).

But despite consuming countless man-hours and numerous fortunes (and prompting at least one kidnapping*) over what is now more than half a century, progress has been glacial. Jet packs are and will probably remain one of the worst under-performing technologies of the post-war era.

"Cusp of coolness" stories are annoying but they can also be dangerous. They give a distorted impression of how technological development works. Columnists and op-ed writers like John Tierney (whose grasp of science is not strong) come away with the idea that R&D is like a big vending machine -- deposit your money and promptly get what you asked for.

It's OK when this naive attitude convinces them to clear out space in their garages for jet packs. It's dangerous when it leads them to write editorials claiming that the easiest way to handle global warming is by building giant artificial volcanoes.


*from Wikipedia:

In 1992, one-time insurance salesman and entrepreneur Brad Barker formed a company to build a rockeltbelt with two partners: Joe Wright, a businessman based in Houston, and Larry Stanley, an engineer who owned an oil well in Texas. By 1994, they had a working prototype they called the Rocketbelt-2000, or RB-2000. They even asked [Bill] Suitor to fly it for them. But the partnership soon broke down. First Stanley accused Barker of defrauding the company. Then Barker attacked Stanley and went into hiding, taking the RB-2000 with him. Police investigators questioned Barker but released him after three days. The following year Stanley took Barker to court to recover lost earnings. The judge awarded Stanley sole ownership of the RB-2000 and over $10m in costs and damages. When Barker refused to pay up, Stanley kidnapped him, tied him up and held him captive in a box disguised as a SCUBA-tank container. After eight days Barker managed to escape. Police arrested Stanley and in 2002 he was sentenced to life in prison, since reduced to eight years. The rocketbelt has never been found.

The Death Arthur Penn and the definition of the Golden Age of Television

(Thought I'd take a break and do some arts and culture blogging. Hell, it's a Sunday.)

Arthur Penn, the stage, television and motion picture director whose revolutionary treatment of sex and violence in the 1967 film “Bonnie and Clyde” transformed the American film industry, died on Tuesday night at his home in Manhattan, the day after he turned 88.

When I saw this obituary in the New York Times a few days ago, it got me to thinking about the Golden Age of Television. Penn made his bones as a director of live dramas and had his big breakthrough with the Miracle Worker which he directed first for television in 1957, then on Broadway and finally on film.

The Miracle Worker was not the only live drama to follow that path. Trip to Bountiful, Visit to a Small Planet and No Time for Sergeants (adapted from the novel by Ira Levin -- yeah, that Ira Levin) also went from television to successful Broadway runs (click here to see a kinescope of Griffith's debut) and were followed by movie adaptations. Others, like Marty and Requiem for a Heavyweight went directly from the small to the big screens.

The reputation of the Golden Age rest largely on a dozen or so of these productions. All of them were remarkable achievements, but if you watch a lot of television from that era you'll probably come away decidedly under-impressed. There were other notable achievements (Caesar and Kovacs stand up well), but on the whole the medium has gotten better.

What has never and will never be matched (though HBO has occasionally come close) was the respect that these these live dramas commanded. These shows were treated like major cultural events, top tier Broadway reviews that opened to an entire nation. The jump to the Tony winning plays and Oscar winning films seemed perfectly natural.

Of course, stage adaptations of TV shows are still with us but now they play to the opposite end of the respectability spectrum.

An experiment in blogging

This will just take a minute of your time.

What follows is a passage from a popular blog, rewritten slightly to make it more general but otherwise unchanged. I'll post the original quote with some comments Monday or Tuesday.

I'd appreciate it if you would take a look at this and give some thought both to the arguments proposed and to the larger belief system they suggest, then come back in a few days and see what effect learning the context has had on your initial impressions.

Thanks.

If you concede that employers need to be able to fire bad employees, then you can't fully defend the role of the unions. You can defend the concept of unions, and you can believe that some of the things unions do, like bargain for higher aggregate wages, help society. But most unions demonstrably make it very difficult to fire bad employees. That is currently a core function of unions, and something that must change. You're also going to need higher salaries to attract a better caliber employee into the workforce, and that's something unions could potentially help. But being "treated like professionals" has to mean both the opportunity to earn a good living if you do well and the potential to be fired if you fail.

I welcome comments but please don't include the source of the passage. Obviously that would undercut the point of the experiment.

Being clear about what the incentives are

Brad Delong discusses the likely result of removing restrictions on insider trading:

If managers free to engage in insider trading know that the next piece of news to be released will cause the stock price to rise, they will buy. If they know that the next piece of news to be released will cause the stock price to fall, they will sell and then buy back later. They don't care whether the news is good or bad--either way they will profit, and either way they will profit equally.

What the ability to engage in insider trading does is that it gives managers an incentive to make the price of the stock vary--they don't care which way. Thus it cannot "serve a useful purpose as an executive compensation device" and cannot "motivate managers to maximize the value of the firm" to shareholders.

Insider trading makes executives' portfolios' long not the company but long the volatility of the company. And shareholders don't want executives making decisions that make the value of companies they own more volatile: stock market investments are risky enough as it is without giving executives reasons to boost the volatility pot.

I think that this really is one of the hardest things about modern economics. The idea of analyzing behavior based on what the incentives are is atremendously powerful tool. The problem is that it is possible to mistake the incentives involved and reach a very poor conclusion.

The part that is scary is that he is actually responding to an actual argument suggesting that permitting insider trading might be a way of incenting executives to maximize the value of a firm.

I am starting to suspect that the analogy of a market as being a structured competition (which I first picked up from reading Joseph Heath) is actually a very useful analogy.

Mommy, where do numbers come from?

One of the nice things about the internet is that, if you look hard enough, you can find people addressing important questions about where statistics (in this case, the plural of statistic) come from.

Over at MacroMania (via Thoma), David Andolfatto talks about how we measure unemployment:

So this is how it works. You are asked in a survey whether you have done any paid work in the previous four weeks. If you answer yes, then you are counted as employed. If you answer no, then you are nonemployed.

If you are nonemployed, you are asked another question: Have you been doing anything in the past four weeks to find work (and if so, what)? If your job search activity is judged to be sufficiently "active," then you are counted as unemployed; otherwise, you are counted as a nonparticipant.

This suggests an interesting point. It would appear that when we talk about job seekers (as in the ratio of job seekers to job openings), we may be excluding the severely underemployed which means things are even worse than they look.

Is this something we should worry about? Damned if I know. That's a question for an econometrician. But it is a good question to ask.

Another good question is who decides on categories and definitions get used when putting together statistics. There have been a lot of pixels spilled about this hypothetical taxpayer receipt put out by the Third Way think tank, but so far as I know, Andrew Gelman was the first to notice this:

I'm more concerned about the list itself, though. I think a lot of cognitive-perceptual effects are involved in what gets a separate line item, and what doesn't. For example, I see the FBI but not the CIA, the NSA, or weapons procurement. There's a line for "salary and benefits for members of Congress" but nothing for the courts system or the White House. And so on. So, while I agree with [Megan] McArdle that "more information is generally better," I'm not quite sure how to get there. I'd be very very suspicious of the choice of items that happens to end up included on the hypothetical itemized tax bill. Especially If it's really true that people don't notice those boxes on their W-2 form with FICA and Medicare payments, I also seem to recall seeing some glossy government documents with charts showing where the money is coming from and where it goes. Maybe there's some place other than a W-2 form to put this information where people will notice it.

The next time you hear a discussion about the fastest way to pump money into the economy...

Tell everybody we have a winner (from the WSJ.com via Thoma):

HOUSTON—At midnight on the first of the month, a scene unfolds at many Wal-Mart Stores Inc. sites that underscores the deep financial strains that many low-income American consumers still face.

Parking lots come to life after 11 p.m. as customers start to stream into the stores, cramming their shopping carts full of milk, infant formula and other necessities.

Then at midnight, when the government replenishes their electronic-benefit accounts with their monthly allotments of food stamps, nutritional grants for mothers with babies or other aid for needy families, they head for the registers.

The airport debate continues

Felix Salmon follows up on his airport post and actually makes matters worse. He asks an expert (Greg Lindsay) what it is that makes the Hong Kong airport good and the response is efficiency. These airports are able to move people quickly and effectively (which, in the end, makes the travel experience more pleasant). But he concludes:

So I’m still not convinced that a major investment in airports is the best — or even a modestly good — use of federal infrastructure-investment funds. Yes, America’s airports are miserable places to travel through. But if what we want to do is boost long-term GDP, then there are better places for the government to spend its money. As and when airports get replaced and upgraded, they will naturally become more modern and efficient. Sadly, however, that’ll take time — and it might not make the passenger experience much better.

I wonder, in a time of low aggregate demand, why investing in more efficient airports is a less favorable form of economic stimulus than building roads or bridges? After all, construction labor is currently cheap and underutilized. Refitting airports so that TSA screening creates fewer queuing problems (for example) or to make them less vulnerable to delays due to poorly designed runways hardly seems like a bad use of infrastructure funds. And it will improve the passenger experience, if only indirectly by reducing the negative externality of lost time due to queuing issues (both of planes and people) that is currently imposed on passengers.

What is the comparison infrastructure investment that is clearly superior? I can think of a few transportation based ones (like improving public transport) but these often lead to new operating costs (running a new bus service, for example). Making airports more efficient might result in lower future costs due to more modern facilities and a better functional design.

What am I missing?

One more from Megan McArdle

In response to a widely circulated itemized breakdown of where your tax dollars go, McArdle writes:

There seems to be an unspoken assumption that opposition to spending rests on misperception of what the money is spent on; Americans tell pollsters they want to cut spending, but it turns out that what they really want cut is the imaginary fortune they think we spend on foreign aid.

But of course, it seems to me that this could just as easily go the other way: isn't it possible that the widespread support for programs like Social Security and Medicare rests on the fact that most people don't realize just how big a portion of your paycheck those programs consume? I don't know the answer to that, but I will point out that most economists believe that paycheck withholdings enable (among other things) higher taxes; if people had to write out a check for their tax bill every year, resistance to income tax increases would be much fiercer.

This suggests that handing people an itemized invoice for their government programs which shows them the total yearly take might increase support for the Smithsonian, and decrease it for the stuff that appears higher up on the bill.

Whichever way it cuts, I think this is a good idea; more information is generally better. I just think that the emotions this sort of receipt provokes in liberal bloggers may not be the same ones it provokes in the average voter.

I think this is a great idea and I'd like to see it implemented immediately. Is it too late to add boxes to our 2010 W2s for Social Security Tax Withheld and Medicare Tax Withheld?

You too can write like a professional pundit -- RMP/STF arguments

This series of post (go here, here, here, here, and while you're at it, here) got me thinking about a rhetorical trick that has gained great popularity in the op-ed pages and other places you find pundits at work. It's called the RMP/STF argument and it's something you can try at home.

Start with a Relatively Minor Problem (RMP) usually expressed through a vaguely troubling statistic. Here are some RMPs:

We might eventually be forced to cut Social Security benefits though not by that much and not for a long time;

A test of international academic performance shows America near the bottom though the test is fairly new and other better established tests and studies contradict this finding;

Airports lack amenities (like plug-ins for laptops) and it can be difficult getting from one terminal to another.

Now pair your RMP with a Serious Tangentially-related Fact (STF). Here are some examples of STFs:

If current trends continue, Medicare will be unsustainable in the near future;

Almost 60% of the students in Nevada will fail to graduate high school;

TSA procedures of questionable value have a large cost to the economy in terms of lost productivity and tourism dollars.

Finally, you propose a solution to your original RMP:

Privatize Social Security;

Fire teachers who don't produce higher test scores;

Reduce zoning restrictions to create more meaningful competition for airports.

Of course, privatizing SS does nothing to help Medicare, reducing zoning restrictions will have no effect on TSA rules and focusing on test scores actually gives schools an incentive to encourage students to drop out, but that doesn't matter. The argument works on an emotional level. It uses the negative associations of the STF to make people more receptive to the solution.

This is closely related to Paul Krugman's "serious person" phenomena where economists, politicians, bankers, etc. are considered responsible and realistic for taking positions directly contradicted by the data. Once again it is the emotional association that matters.

When you criticize an RMP/STF argument, someone will invariably say "At least, _____ is willing to talk about [serious issue]." That's not really true. _____ is willing to invoke the issue and we need a higher standard than that.