Saturday, November 19, 2011

Evaluating evidence

I want to steal a quote from Paul Krugman to illustrate a point:

And in the end, Ryan’s answer is that we need strong economic growth, the kind that we get by cutting taxes on the rich. Because that’s why the Clinton years were an economic disaster and the Bush years so prosperous.

Is this strong evidence?

First of all, we need to consider a number of causal hypotheses:

1) Tax rates on the rich are unrelated to economic growth
2) Higher tax rates on the rich increase economic growth
3) Economic growth makes it easier to tax the rich
4) Higher tax rates on the rich decrease economic growth

Then we need to consider lags between tax policy and changes in economic growth. I am suspicious of anyone who says that this is an easy problem. After all, what we really want (the counterfactual of what would happen if Bush/Clinton not changed tax policy) is completely unavailable.

So what value is this evidence?

It does rule out one very clear talking point in the debate. It suggests that moderate changes in tax policy (Bush Tax cuts) do not have a stronger effect on economic growth than the economic fundamentals do. We may even take this as weak evidence of hypothesis #4 above (with all of the caveats about not being able to make a strong inference).

So the ideas that tax cuts [focused on high income earners] are a good response to short term problems with weak economic growth seems to be contrary to the best evidence available. Nor does looking at period like the 1950' (with very high marginal rates and rapid growth) seem to provide a lot of support for Hypothesis #2.

But if it is case that Hypothesis #2 is true, we know that it is unlikely to overcome other economic issues (or it would have made the Bush years a time of prosperity). Or, in other words, that the overall effect size of this tax policy change is small relative to other factors (if it works in the direction predicted by Hypothesis #2). Now one can reframe this as a moral question, and some do.

But it is worth considering that, in the absence of controlled experiments, how do we update our expectations when a strategy that sounds reasonable doesn't seem to give expected results.


  1. This reminds me of a point I often make to grad students: if you have a sample that is too small to confirm a hypothesis or fit a model, it can still be enough to allow you to reject some simple hypotheses which, in the absence of that small dataset, might otherwise seem potentially plausible.

  2. Andrew,

    What's discouraging is how seldom this common sense principle is applied to the public discourse.

  3. It's a common-sense principle but I've never seen it formalized or put in any book.

  4. Agreed. I think that these inference approaches are especially important in data with high importance but low numbers. For example, I wonder if presidential election forecasting would not run into the same problems?

  5. One thing that comes up in election forecasting is: What level of confidence do you want with for your national forecasts? I don't think 95% intervals make sense. Why? Because 20 elections will take 80 years. In 80 years you'd expect a (statistical) regime change of some sort. So I think it doesn't make much sense to ask for that level of confidence in a mechanical forecast.

  6. Mike Kimmel at Angry Bear has done the obvious regressions using 20th century data, and no matter how he slices it, peak marginal rates around 60-70% maximize economic growth. He's tried to eliminate confounding factors, and I imagine his confidence level is low, but sometimes you have to work with the data you have. People don't drink gasoline based on less data than that.

  7. It would be nice if Mike Kimmel's work was to make it into the national debate. The idea that we are willing to SACRIFICE economic growth to have a low marginal tax rate on high earners would certainly change the color of the debate.