The perils of forecasting

Mark points out, in a comment on my post on financial literacy, the issue of the equity premium as being discussed by Felix Salmon. I think that this point is really interesting and deserves more attention than a breif comment.

This article from the Economist is another great discussion of the complexities of making this claculation. The author lays out the basic problem with calculating an equity premium (or lack thereof) using current data (i.e. recent rates of return). One does not want to price in the current economic crisis (unless one thinks that these crises will be more common in the future) but neither does one want to calculate an equity premium that ignores either key events or actual real rates of return.

When you add in the issue of limited data (we only have a couple of hundred years of annual stock returns for the United States and only about 60-80 years are really relevant to the current market) and the risk of a secular shift (what if one or more financial innovations has fundementally changed the nature of the market) then these complications are almost enough to make the problem intractable. One might argue that these factors need to balance out in the long run (if equities don't pay a premium for risk then people will stop holding equities). But it is remarkable how long the long run can be; I think John Maynard Keynes said it best with:

The long run is a misleading guide to current affairs. In the long run we are all dead. Economists set themselves too easy, too useless a task if in tempestuous seasons they can only tell us that when the storm is past the ocean is flat again.

I think that this problem is true for epidemiological forecasts as well. The forecast of influenza rates (as a recent example) depends critically on the assumption that the current strains of influenze are not fundementally different than past strains. Often this assumption is reasonable but it can miss the most important changes (like the arrival of a new and more lethal version of the virus).

Whether it is disease rates or stock markets, it is not a simple matter to use the past as a guide to the future. There is no doubt that forecasting is hard but it's also true that it is important to do it as well as possible. If I ever figure out the trick I will be sure to share it!