Evaluating evidence

I want to steal a quote from Paul Krugman to illustrate a point:

And in the end, Ryan’s answer is that we need strong economic growth, the kind that we get by cutting taxes on the rich. Because that’s why the Clinton years were an economic disaster and the Bush years so prosperous.

Is this strong evidence?

First of all, we need to consider a number of causal hypotheses:

1) Tax rates on the rich are unrelated to economic growth
2) Higher tax rates on the rich increase economic growth
3) Economic growth makes it easier to tax the rich
4) Higher tax rates on the rich decrease economic growth

Then we need to consider lags between tax policy and changes in economic growth. I am suspicious of anyone who says that this is an easy problem. After all, what we really want (the counterfactual of what would happen if Bush/Clinton not changed tax policy) is completely unavailable.

So what value is this evidence?

It does rule out one very clear talking point in the debate. It suggests that moderate changes in tax policy (Bush Tax cuts) do not have a stronger effect on economic growth than the economic fundamentals do. We may even take this as weak evidence of hypothesis #4 above (with all of the caveats about not being able to make a strong inference).

So the ideas that tax cuts [focused on high income earners] are a good response to short term problems with weak economic growth seems to be contrary to the best evidence available. Nor does looking at period like the 1950' (with very high marginal rates and rapid growth) seem to provide a lot of support for Hypothesis #2.

But if it is case that Hypothesis #2 is true, we know that it is unlikely to overcome other economic issues (or it would have made the Bush years a time of prosperity). Or, in other words, that the overall effect size of this tax policy change is small relative to other factors (if it works in the direction predicted by Hypothesis #2). Now one can reframe this as a moral question, and some do.

But it is worth considering that, in the absence of controlled experiments, how do we update our expectations when a strategy that sounds reasonable doesn't seem to give expected results.

Challenges of causal inference in complex systems

From Felix Salmon:

“Spend less money, create more jobs” is the kind of world one normally finds only in Woody Allen movies, and it’s a profoundly unserious stance for any politician to take. Spending cuts, whether they’re implemented by the public sector or the private sector, are never going to create jobs. And there’s simply no magical ju-jitsu whereby government spending cuts get reversed and amplified, becoming larger private-sector spending increases.

I think that one of the difficulties in macroeconomics is that you have complex systems that are not subject to experimentation. So you are forced to try and use observational studies and analogies with microeconomics to try and determine the causal effects of policies. Even instruments are questionable as they also rely on unverifiable, strong assumptions.

The inability to have a consensus on the counter-factual is pernicious and causes no end of trouble. Consider the tax increases passed at the beginning of the Clinton administration. Are they responsible for the late-1990's boom, unrelated to it, or did they act to slow it down (making the current economy smaller than it could have been)? How would you know this?

Cross country comparisons are possible but you have both confounding factors and effect measure modification. Changing the tax rate in Sweden might have different consequences than in the United States due to both different cultures (confounding) and to differences in current tax rates (effect modification). So, by picking different analogies and different models for the observational data, we can end up with some really strange claims being made about how economies work.

It is not an area with easy solutions. But I think to agree with Felix that the model he is critiquing is making heroic assumptions about the influence of tax levels on economic growth.

Instrumental Variables

Reading this post by Andrew Gelman got me thinking: how do we really know if a variable is an instrument? Demonstrating that a variable is an instrument often seems to be a matter of telling a compelling clinical story but leaves one with strong and unverifiable assumptions.

This is a real issue with the Physician Preference instrument in pharmacoepidemiology where it has the potential to be either a major advance or a blind alley. But how would one know for sure? I suspect that it is more likely the former than the latter but sorting this out with any level of certainty isn't easy.

Any insights out there?