Perils of studying Small Effects

From Andrew Gelman:

For example, suppose you're a sociologist interested in studying sex ratios. A quick review of the literature will tell you that the differences in %girl births, comparing race of mother, age of mother, birth order, etc, are less than 1%. So if you want to study, say, the correlation between parental beauty and sex ratio, you're gonna be expecting very small effects which you'll need very large sample sizes to find. Statistical significance has nothing to do with it: Unless you have huge samples and good measurements, you can pretty much forget about it, whether your p-value is 0.02 or 0.05 or 0.01.

I think that this is correct and ties into a larger narrative: there are a lot of small effects of great theoretical interest that are really hard to show in actual data due to the limitations of realistic sample sizes. Observational epidemiologists often try to handle this by looking at prescription claims databases (for example). But these studies create a new problem: there is a serious concern of unmeasured confounding due to factors that are not measured in these data sources (e.g. smoking, alcohol use, diet, exercise). It's not really a big step forward to replicate lack of power with potential bias due to confounding.

I think the real issue is simply that small effects are difficult to study, no matter how interesting that they are. So I think that Andrew Gelman is right to call for the interpretation of these studies to done in the context of the larger science.

Yikes!

This is pretty bad. I was surprised at just how inexpensive the organ transplant program really was . . .

Nice Observation

Matt Ygelsias makes a good point:

Today’s report, which is being called bad numbers, we learned of 50,000 new private sector jobs and the loss of 11,000 more public sector jobs.

To me, this really does seem like a bad result. But conservatives in the audience need to recognize that we’re getting what they say they want. The private sector is growing and the public sector is shrinking. If you’re disappointed in the results, you can perhaps consider reviving your view of the desirability of executing this shift in the middle of a collapse in aggregate demand. But what you can’t do is say that the bad economy is somehow being caused by an Obama-era slide toward socialism. The slide isn’t happening. The explosion in government isn’t happening. The public sector is shrinking.

The precise balance between public and private employment is tricky. But I think it is good to recognize what the actual direction of change is.

Lottery based admissions

From the Harvard Crimson:

For one thing, “excellence” in the Harvard admissions process—and at Harvard—has a lot less to do with virtuous character traits than with an ability to game the system. By placing a premium on students who go above and beyond in extracurricular realms, Harvard has attracted a number of truly incredible people but has also encouraged a high school arms race wherein kids cram their schedules with activities in an attempt to attract admissions officers.

By selecting for this kind of behavior, the admissions process doesn’t encourage real excellence, but, to use the novelist Walter Kirn’s term from his hilarious book and essay “Lost in the Meritocracy,” “aptitude for showing aptitude.” This may well be of use in students’ careers after college, but it is orthogonal if not antithetical to the goals of a liberal arts education.

I think that these factors are also an issue when you evalaute education at any level. People compete hard to get into top schools -- the same skills that they use to "game the system" also tend to ensure decent post-Harvard outcomes.

This makes evaluating the actual contribution of the education at Harvard, itself, tricky (to say the least).

h/t: Felix Salmon

More on owning preperty to vote

Commenter David made the point:

This criticism always sounds a little hollow--when people talk about interpreting the constitution 'as the framers intended', I'm pretty sure they don't mean the parts that later amendments changed.

What is difficult is that it is pretty clear from his quote that Judson Phillips really does believe this; go look. Now it could be that this is a minority view and not very typical.

But I jumped on it because this comment seemed to open a lot of issues:

But one of those was you had to be a property owner. And that makes a lot of sense, because if you’re a property owner you actually have a vested stake in the community.

So what happens if you have a mortgage on a property? Do you own it? What about a lien? How do you easily keep track of who has successfully sold their house (and thus lost the right to vote). When does this count? What if you buy a house right before election day?

Furthermore, how much property is enough? Do we value it by space? What about by value? In a country as diverse as the United States, either Wyoming or Manhattan is going to be complicated.

How would this interact with eminent domain? If the state seizes your house are you disenfranchised? Could this interact in unexpected ways with the idea of equal protection?

If you own land in city X but live (by renting) in city Y, where do you vote? What if you rent your house but own a cottage? Or a garden plot?

Finally, I wonder if the act of owning land really leads to stability given how liquid houses are. It might become more of a marker of wealth; arguing that only the wealthy should vote is a dangerous position.

So I was mostly struck by how badly this proposal seemed to be thought out and how complicated it would be to implement.

What happens if you have a mortgage?

Jon Chait talks about a new idea from Tea Party Nation:

Tea Party Nation Judson Phillips thinks the franchise should be taken away from renters

Of course, the complexities in trying to operationalize such a plan are, shall we say, staggering.

Higher education and resource allocation

It is a powerful argument to reduce the concerns of your opposition by reframing their views into something that seems absurd.

Dean Dad writes:

Administrators may be the bearers of bad news, and sometimes the people who have to choose among terrible options. But to assume that we’re sitting on piles of money, cackling with glee while exploiting adjuncts and pocketing the savings for ourselves, is just otherworldly. It assumes a context completely out of keeping with anything I can recognize as reality. It’s so far afield that the only truly fitting rebuttal is a sigh.

I think that the issue is not that anybody is sitting on piles of cash. I think that there are two legitimate concerns. One, is the redistribution of resources (which is always going to be a feature of any organization that is not experiencing enough growth for all parties to "win" simultaneously). Two, is the concern about the growth of adminsitrative costs. Some of this growth is beyond the control of the administration (unfunded federal reporting mandates come to mind). But I think that it is a fair position to want to open a dialogue on this issue.

To dismiss these concerns out of hand doesn't seem to be the best way to encourage an open dialogue.

All costs are linked

From Austin Frakt:

It’s convenient to think about health care costs and expenditure organized in two sectors, public and private. But one deceives oneself in thinking they’re independent. A principle problem with control of public health care spending is that public prices can’t deviate too far from private ones. If they do, Medicare beneficiaries will experience access problems, as Medicaid beneficiaries already do.

I think that this is entirely correct. Presuming that the same procedure is being done, it is reasonable to expect that people would prefer to be paid more. When the cost differential is low then it isn't worth putting a lot of effort into trying to attract the higher yield business. But as soon as prices deviate enough, you would expect doctors to compete on non-financial grounds. Speed of access comes immediately to mind.

I still find it odd that the United States socializes the medical care that is the most expensive (end of life care) and not that of inexpensive groups (like 20-something adults). Perhaps that is simply needed due to issues of poverty and ill health in old age. But these sorts of tensions cannot be good.

Tyler Cowen appears to agree:

The differential payment rates across Medicare, Medicaid, and private insurance are becoming unsustainable more quickly than I had anticipated; see for instance the link in #4. Further reforms will be required more quickly than had been anticipated, but it's not obvious how such reforms should proceed. It's hard to either upgrade the Medicaid (and Medicare) rates or to downgrade the private insurance rates. Monitor this one closely, because it is likely to prove the breaking point of our health care status quo, with or without the Obama plan. (This is our version of the ticking time bomb within the eurozone, namely that natural rates of growth split apart a distortion, increasingly, over time.)

I think that these arguments mean that private health care costs will move back into the debate sooner rather than later. Add in the fast growth of health insurance costs and you can immediately see that pressure is going to come from many directions.

Of course, the idea of the Bowles-Simpson Deficit Plan to tax health insurance plans would actually accelerate the tensions here by making private plans even less affordable (which could have a huge impact on the lower middle class).

It's a complicated system with a lot of moving parts. But I think economists are right to worry about these costs diverging as much as they currently are.

A point to ponder

From Worthwhile Canadian Initiative:

Indeed, this is the fatal attraction of socio-biology - it's too easy to come up with untestable explanations for just about anything.

I think that this quote is exactly right. A lot of really interesting stories can be told that are not empirically testable. This approach can lend a veneer of respectability and scientific rigor to what is merely a form of "educated guessing".

Comments on Grants

Professor in Training mentions here recent experience of good comments and yet bring triaged by a granting agency. I think that this type of triaged is at least partially a consequence of the modern funding environment. A lot of people work very hard to submit good ideas and well written grants. As paylines shrink (and the endowments of a lot of foundations shrink due to market turmoil), I suspect that many fundementally sound research proposals will suffer this fate. It no longer needs to be great, it needs to be actively exciting.

Or maybe the real comments are coded. But I think the former suggestion of limited funding has a lot of explanatory power.

"The Instability of Moderation"

Some essential Krugman today. You may not agree that we've entered the Dark Age of macroeconomics but Krugman makes a good case for how and why things got so bad:

The brand of economics I use in my daily work – the brand that I still consider by far the most reasonable approach out there – was largely established by Paul Samuelson back in 1948, when he published the first edition of his classic textbook. It’s an approach that combines the grand tradition of microeconomics, with its emphasis on how the invisible hand leads to generally desirable outcomes, with Keynesian macroeconomics, which emphasizes the way the economy can develop magneto trouble, requiring policy intervention. In the Samuelsonian synthesis, one must count on the government to ensure more or less full employment; only once that can be taken as given do the usual virtues of free markets come to the fore.

It’s a deeply reasonable approach – but it’s also intellectually unstable. For it requires some strategic inconsistency in how you think about the economy. When you’re doing micro, you assume rational individuals and rapidly clearing markets; when you’re doing macro, frictions and ad hoc behavioral assumptions are essential.

So what? Inconsistency in the pursuit of useful guidance is no vice. The map is not the territory, and it’s OK to use different kinds of maps depending on what you’re trying to accomplish: if you’re driving, a road map suffices, if you’re going hiking, you really need a topo.

But economists were bound to push at the dividing line between micro and macro – which in practice has meant trying to make macro more like micro, basing more and more of it on optimization and market-clearing. And if the attempts to provide “microfoundations” fell short? Well, given human propensities, plus the law of diminishing disciples, it was probably inevitable that a substantial part of the economics profession would simply assume away the realities of the business cycle, because they didn’t fit the models.

The result was what I’ve called the Dark Age of macroeconomics, in which large numbers of economists literally knew nothing of the hard-won insights of the 30s and 40s – and, of course, went into spasms of rage when their ignorance was pointed out.

Political instability

It’s possible to be both a conservative and a Keynesian; after all, Keynes himself described his work as “moderately conservative in its implications.” But in practice, conservatives have always tended to view the assertion that government has any useful role in the economy as the thin edge of a socialist wedge. When William Buckley wrote God and Man at Yale, one of his key complaints was that the Yale faculty taught – horrors! – Keynesian economics.

I’ve always considered monetarism to be, in effect, an attempt to assuage conservative political prejudices without denying macroeconomic realities. What Friedman was saying was, in effect, yes, we need policy to stabilize the economy – but we can make that policy technical and largely mechanical, we can cordon it off from everything else. Just tell the central bank to stabilize M2, and aside from that, let freedom ring!

When monetarism failed – fighting words, but you know, it really did — it was replaced by the cult of the independent central bank. Put a bunch of bankerly men in charge of the monetary base, insulate them from political pressure, and let them deal with the business cycle; meanwhile, everything else can be conducted on free-market principles.

And this worked for a while – roughly speaking from 1985 to 2007, the era of the Great Moderation. It worked in part because the political insulation of central banks also gave them more than a bit of intellectual insulation, too. If we’re living in a Dark Age of macroeconomics, central banks have been its monasteries, hoarding and studying the ancient texts lost to the rest of the world. Even as the real business cycle people took over the professional journals, to the point where it became very hard to publish models in which monetary policy, let alone fiscal policy, matters, the research departments of the Fed system continued to study counter-cyclical policy in a relatively realistic way.

But this, too, was unstable. For one thing, there was bound to be a shock, sooner or later, too big for the central bankers to handle without help from broader fiscal policy. Also, sooner or later the barbarians were going to go after the monasteries too; and as the current furor over quantitative easing shows, the invading hordes have arrived.

Orlando Airport

It has been a surprisingly pleasant experience, so far. The TSA screening was fast and polite (and the screener making announcements made public service messages witty). One can only hope Canadian customs is as pleasant.

But this has been a good start, at least.

Happy Thanksgiving

It has been a good year for OE (the default abbreviation of the blog title). We started in March 2009 as an experiment. The addition of Mark to the blog list has been a big improvement in the quantity and quality of posts. In 2010 we have months with more blog posts than the 2009 total.

So I hope that you all have a safe and Happy Thanksgiving!

Dental Care

So it looks like I am getting my second root canal next week due to a filling that went too deep. It is, sadly, one of my front top teeth (which makes the "just pull it" option unappealing).

There has been a lot of talk of health care, per se, at sites like the Incidental Economist and Marginal Revolution. But I am finding dental care to be shockingly expensive even with good dental insurance and a reasonable salary. Having experienced severe tooth pain (repeatedly), I wonder what happens to people who cannot afford care.

I will also say that free market options are limited when you are in blinding pain with a tooth abscess. One is willing to pay almost anything to make the pain stop and worry about things like rent later. Certainly it is a poor time for comparison shopping. Nor does the fact that many patients pay for most of their care seem to do anything to control prices.

These episodes have made me increasingly tough on my dental hygiene (flossing, brushing, ultrasonic toothbrush, prescription tooth paste). But I fear that some aspect of my diet or lifestyle still seems to be out of whack because dental decay appears to be accelerating at an alarming pace.

But it does make me wonder why dental care is treated differently than, for example, back pain.

Industrial Policy

I am a huge believer in mixed economies. But it is hard to prove that they are the best option for exactly the reasons that Ms. McArdle is skeptical about industrial planning in China:

It's not that I didn't understand that the government did this; it's that I didn't understand how pervasive it would be, or how popular this would be, at least with the folks we interview. Everyone--including most of the economists and NGOs--seems to think this is swell. No fiddling around with archaic, unplanned systems; just figure out what the country needs and do it!


Perhaps it is just my ideological blindness that makes me believe that this cannot, in the long run, turn out well. But there's a plausible story that the early boom was mostly a matter of removing distortions (and taking advantage of capital, human and otherwise, accumulated in Hong Kong and China). Now the government is much more directly picking winners and losers. They're not trying to manage growth; they're trying to cause it in places where it shows little sign of happening organically.


It's not that I think that no form of industrial policy can ever have good effect. Can government build infrastructure to good effect? Yes, certainly. Can they manage growth? Can it occasionally pick industrial winners? They have in the past--though on average, I'd say it's abundantly clear that governments have more often picked, and sustained, losers. And the more comprehensive the industrial policy, the worse the economic losses have generally been.

The issue here is that successes can be explained by a lot of different factors (as can failures). And it is pretty clear that people have strong "priors" (in the Bayesian sense) for the approach that they favor. This makes it hard to use the (limited in scope) data to decide between approaches. Add in confounding factors and it gets harder. Add in changes in technology and you guarantee issues.

Just consider, for example, stock market returns. What is the relevant period of interest? Some people claim you can consider returns on stock from 1800 to today based on what records are available. But clearly the information available to stock analysts is different today than in 1960. Now consider that you want to look at long term returns (i.e. saving for retirement) and suddenly the noise threatens to overwhelm the data -- as you really have only a very few (tightly correlated) time trends.

How much worse is that for looking at economic growth?

No wonder these arguments are difficult to make . . .