Homework and treatment levels

Previously on OE, we've talked about how norming, placebo and volunteer effects can cause methods like lottery-based analyses to miss significant sampling effects. Here's one more area of potential concern.

The basic problem here is familiar to clinical researchers. You're trying to determine the optimal level of a drug or therapy or exercise regime or type of food. There is an ideal dosage, the level of treatment you would prescribe if your only concern was optimizing the effect. Then there is what we might call a realistic dosage, one that takes into account factors like side effects and compliance. These levels vary from group to group which means the process of finding the right dosage is sensitive to sampling issues. The appropriate level is likely to be different for a group of college students and a group of senior citizens.

Teachers assigning homework face essentially the same problems. Assuming that goal is to improve students' score on a standardized test, there is an ideal homework assignment for each lesson, a subset of the problems available in the text that will tend to maximize the average score of a class (to keep things simple we'll limit the discussion, with no loss of generality, to homework assignments derived from the textbook).

The ideal assignment assumes that students have unlimited time and will complete all the assigned problems. The realistic assignment would take into account factors like time constraints, demands from other classes, compliance, burnout and parental pushback (I can tell you from experience that parents complain if they feel their children are being given excessive work and I would argue this is a good thing -- teachers should remember that their students' time also has value). The realistic assignment would optimize the class's average score when these constraints are in place.

Like the previously discussed norming, placebo and volunteer effects, a sampling/treatment level interaction with the homework assignment here can interfere with the ability of lottery-based analysis to detect sampling effects.

To see how this would work, consider a charter school like one in the KIPP system that very publicly acknowledges that students will be expected to do large amounts of homework each night (some schools even require parents to sign contracts agreeing to sign off on the students' homework). Students who apply for this school are aware of these requirements, as are their parents. This certainly raises the possibility that the optimal realistic homework level might be higher for these students, particularly if this is one of those schools that aggressively culls out the students who can't handle the workload.

If this is the case and if both the charter school and the public schools assign the optimal levels of homework for every class, you will have a sample based effect that will completely evade detection by a lottery-based analysis. That analysis will compare the performance of those who applied for the charter and were accepted against those who applied but lost the lottery. Since the rejected students will receive a treatment level that was optimized for the general population while the accepted students will receive a treatment level optimized to their particular subgroup, we expect the charter school students to do better, leading the lottery-based analysis to incorrectly conclude that there is no selection effect.

I used homework in this post to keep things simple but the principle applies just as well to most elements of the educational models of many highly praised charter schools -- longer days, Saturday instruction, extended school years, holding students responsible for more material on tests. Put bluntly, these schools get their results by giving students lots of work. Because of these results, these schools are often held up as examples for the rest of the educational system, but when you start with a self-selected sample then counsel out the students in that sample who have trouble keeping up, you should at least consider the possibility that the optimal workload for your student body is higher than the optimal workload for the general populace.

This doesn't mean that all high performing charter schools are simply running on undetected selection effects and it certainly doesn't mean that charter schools with high standards should lower them. What is it does mean is that selection effects in education show up in subtle and complex ways. It remains extraordinarily difficult to resolve these issues through observational means. Any study that claims to have settled them should be approached with great caution.

Arnold Kling

Arnold Kling is a skeptic about the importance of peer effects:

Back when the SAT was just math and verbal, I described us as living on the other side of a 250-point SAT gradient--in the better high schools, the SAT's were 1200, and in our high school they were 950. (I took the view that my own kids did not need high-achieving peers in order to do well on their own.)

I am not sure whether this is persuasive (although it does match my personal experience). But it is true that this is the sort of behavior that I would expect from peer-effects skeptics (and if we saw more of it there would be a much smaller premium on good schools for housing costs.

Thomas Jefferson on income inequality

From a letter to James Madison (courtesy of Brad DeLong):

I am conscious that an equal division of property is impracticable, but the consequences of this enormous inequality producing so much misery to the bulk of mankind, legislators cannot invent too many devices for subdividing property, only taking care to let their subdivisions go hand in hand with the natural affections of the human mind. The descent of property of every kind therefore to all the children, or to all the brothers and sisters, or other relations in equal degree, is a politic measure and a practicable one. Another means of silently lessening the inequality of property is to exempt all from taxation below a certain point, and to tax the higher portions or property in geometrical progression as they rise.

Whenever there are in any country uncultivated lands and unemployed poor, it is clear that the laws of property have been so far extended as to violate natural right. The earth is given as a common stock for man to labor and live on. If for the encouragement of industry we allow it to be appropriated, we must take care that other employment be provided to those excluded from the appropriation. If we do not, the fundamental right to labor the earth returns to the unemployed. It is too soon yet in our country to say that every man who cannot find employment, but who can find uncultivated land, shall be at liberty to cultivate it, paying a moderate rent. But it is not too soon to provide by every possible means that as few as possible shall be without a little portion of land. The small landholders are the most precious part of a state.

Profundity adjacent

I'm not a fan of the BBC's the Changing World. The show has interesting topics but the episodes I've caught have an annoying habit of almost saying something profound and insightful then veering off at the last minute. This tendency to be profundity adjacent is particularly notable because, on KPCC, Changing World airs a couple of hours after This American Life, a show with an extraordinary ability to uncover the genuinely important aspects of a complicated story.

The criticism certainly holds for this episode on chess. Repeatedly, I found myself caught up in an interesting thread only to have it end right before it got to the good part. Still, they were interesting threads and if your interests include abstract strategy games and their role in fields like education, you should probably check this out.




http://www.thechangingworld.org/archives/2010/44.php

On Salmon's blog, even the comments are quotable

Felix Salmon has a good post on the economic impact of extending unemployment benefits but for me the most insightful observation was found in the comments:

What’s missing from this discussion is the part about optimization of deployment of resources. If a skilled worker, say a skilled aluminum welder, can stay on unemployment long enough to find a skilled aluminum welder job at higher pay and productivity, that’s good for GDP and has a higher multiplier. But if that worker has to settle for a security guard job because he or she must pay the rent, then there is a net loss of productivity, GDP, and related multipliers.

Wargaming

This article is a very interesting look at the rise and fall of wargaming. Of especial interest was:

In 1982, the 30,000 subscribers of Strategy & Tactics, SPI's flagship magazine, were the most avid wargame enthusiasts on the globe. More than 50% of them, per SPI's own feedback, owned 100 or more wargames; most of them bought a dozen or more games every year, not counting the games they received as subscribers to the magazine. SPI estimated that perhaps 250,000 people, in the whole of North America, had ever bought a wargame; and the 30,000 subscribers to S&T bought an enormously disproportionate number of the games sold.

They were the hard core, the fundament upon which the whole wargame industry was built.

So naturally, when TSR took over SPI, the first thing it did was give the finger to S&T's subscribers. The first thing it did was say to the best customers of its new subsidiary, "Go take a hike; we don't want your custom; your concerns mean nothing to us."

You see, almost every magazine in the country can declare bankruptcy tomorrow, if it wants to, because every magazine in the country has enormous liabilities: the obligation to provide issues to its subscribers. The subscription money came in long ago, and has been spent, and that liability remains. Magazines continue, and make money, because they sell advertising, and expect their subscribers to resubscribe. But SPI had no money -- and S&T had 30,000 subscribers. Over a thousand were life-time subscribers, owed issues in perpetuity in exchange for no further income.

TSR didn't want the liability. Fulfilling S&T's existing subscriptions would have cost it money.

So, TSR decided, it would not honor any subscriptions.

TSR had taken over SPI's assets, but not its liabilities, so they claimed; therefore, they had no obligation to S&T's subscribers.

Gosh. Guess what happened? Few of S&T's subscribers reupped. Few wanted to send more money to the company that had just ripped them off.

And few of them ever bought any of the wargames TSR began to publish.

And TSR never could figure out why their wargames never sold.

I think that this example is very interesting for a couple of reasons. One, the fact that this was such a large setback to the entire field makes it clear just how important barriers to entry can be in making competition difficult (it would have been infeasible to develop the network that had just been destroyed by another publisher). Two, is the market power of corporations. With their ability to raise equity, they can purchase viable business models but they do not necessarily manage them well.

For a luxury good, like wargames are , it is unfortunate for people who enjoyed them but hardly a tragedy -- other goods will have the chance to thrive in the market instead (and perhaps this is a form of creative destruction?).

But imagine this process for an essential good: like water, electricity or education. I think we should keep examples like this in mind when considering the benefits of privatization. There are massive upsides to capturing the efficiency of a functioning market -- but only if players are allowed to fail.

Funding Cuts

From Dead Dad:

Of course, that refers only to general college reserves. It’s also common for various programs to have reserves of their own, earmarked for specific purposes. The college foundation might have reserves dedicated for certain scholarship awards. Some grant-funded programs will have reserves for specific functions and only for those functions. (In the context of multiyear grants, for example, it’s common to have ‘carryover’ of excess funds from one fiscal year to the next. That’s frequently allowed, but that doesn’t give license to transfer the extra grant money to the general college budget.) In cases like those, money comes with strings attached, and violating the terms of the money involves forfeiting the money. You can’t just move it around.

This lack of flexibility actually highlights one of the difficult issues with cuts in grants. If grants are cut by 10%, you don't have the discretation to eliminate 10% of the ongoing projects to make sure that the rest are successful. In the same sense, going after indirects to try and make up these losses would lead to the defunding of other operations.

The modern university budget looks remarkably inflexible to me, which makes planning for adverse financial circumstances appear to be extremely tricky. Not only do you have to cut but what you can cut can be remarkably constrained. This can lead to very poor optics (where something that looks non-essential is fully funded whereas a core operation simply lacks funding).

It is a tough place to be in!

Education reform and the two kinds of knowledge

"Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information upon it. When we enquire into any subject, the first thing we have to do is to know what books have treated of it. This leads us to look at catalogues, and at the backs of books in libraries."
— Samuel Johnson

This came up while I was working on a longer piece but I decided it deserved a short post of its own.

In the two and a half centuries since Dr. Johnson's time, the amount and the value of the second kind of knowledge has increased substantially (as it had in the preceding centuries). Cheap printing, the internet and telecommunications (consulting an expert qualifies as the second kind) have given us ready access to a stunning array of information.

With a few exceptions, the ability to find and manipulate information is far more important than the ability to simply repeat it, but the push for accountability in education often focuses strictly on the first kind.

update: You can find a related discussion here.

Someone finally notices the orphan technology

As I mentioned before, over-the-air broadcasting has been a conspicuous hole in any number of media stories, probably due to the demographics of that market and the lack of interested parties pushing the story. Eventually though, once the market included enough young, photogenic professionals, the New York Times did pick up on the story (from this Sunday):

Many pay TV customers are making the same decision. From April to September, cable and satellite companies had a net loss of about 330,000 customers. Craig Moffett, a longtime cable analyst with Sanford C. Bernstein, said the consensus of the industry executives he had talked to was that most of these so-called cord-cutters were turning to over-the-air TV. “It looks like they’re leaving for the antenna,” he said.

The article has its problems. There are factually challenged passages like this:

The new antennas do pull in more programs than your grandfather’s rabbit ears, because of new channels that broadcasters added during the transition to digital signals. The broadcasters can fit multiple digital channels into the same frequencies that used to carry one analog channel.

In St. Paul, for example, where Ms. Bayerl lives, there are extra channels from ABC and NBC with local news and weather, four public television channels and a music video channel. Big markets like Los Angeles have 40 or more channels, according to Nielsen.

(I live in LA county and, according to my digital converter I get 114 channels. There are at least a half dozen more I could get if I wanted to invest in a better antenna set-up. When you're off by a factor of three, "or more" just doesn't cut it.)

The article also completely overlooks the role that the Hispanic market plays in the over-the-air broadcasting story which is a bit like ignoring the role commuters play in the NPR audience.

Still, compared to what we've seen up till now, this account is almost cutting edge.

Update:

While picking up a printer cartridge this morning I fact-checked the following passage from the NYT story:

Modern antennas, which cost $25 to $150, pick up high-definition signals that can actually be crisper than the cable or satellite version of the same program, because the pay TV companies compress the video data.

Actually HDTV compatible antennas the one I use for my 114 channels are available at Target start at $10.98.

NIH news

From Medical Writing, Editing & Grantsmanship:

According to the AAMC, seems most likely these CRs will be the high-point of FY11:

During debate on the measure, Rep. Jerry Lewis (Calif.), the ranking Republican on the House Appropriations Committee, again expressed opposition to “any potential omnibus spending bill the Democratic leadership may be planning to bring to the House floor before the end of the year.” He also opposed extending the CR for the balance of FY 2011 at current level, which he described as “frankly, too darn high.”

I won't argue the merits or deficits of this view at the policy level except to say that, from a personal point of view, this news doesn't help my plans to start a career in this area.

Perils of studying Small Effects

From Andrew Gelman:

For example, suppose you're a sociologist interested in studying sex ratios. A quick review of the literature will tell you that the differences in %girl births, comparing race of mother, age of mother, birth order, etc, are less than 1%. So if you want to study, say, the correlation between parental beauty and sex ratio, you're gonna be expecting very small effects which you'll need very large sample sizes to find. Statistical significance has nothing to do with it: Unless you have huge samples and good measurements, you can pretty much forget about it, whether your p-value is 0.02 or 0.05 or 0.01.

I think that this is correct and ties into a larger narrative: there are a lot of small effects of great theoretical interest that are really hard to show in actual data due to the limitations of realistic sample sizes. Observational epidemiologists often try to handle this by looking at prescription claims databases (for example). But these studies create a new problem: there is a serious concern of unmeasured confounding due to factors that are not measured in these data sources (e.g. smoking, alcohol use, diet, exercise). It's not really a big step forward to replicate lack of power with potential bias due to confounding.

I think the real issue is simply that small effects are difficult to study, no matter how interesting that they are. So I think that Andrew Gelman is right to call for the interpretation of these studies to done in the context of the larger science.

Yikes!

This is pretty bad. I was surprised at just how inexpensive the organ transplant program really was . . .

Nice Observation

Matt Ygelsias makes a good point:

Today’s report, which is being called bad numbers, we learned of 50,000 new private sector jobs and the loss of 11,000 more public sector jobs.

To me, this really does seem like a bad result. But conservatives in the audience need to recognize that we’re getting what they say they want. The private sector is growing and the public sector is shrinking. If you’re disappointed in the results, you can perhaps consider reviving your view of the desirability of executing this shift in the middle of a collapse in aggregate demand. But what you can’t do is say that the bad economy is somehow being caused by an Obama-era slide toward socialism. The slide isn’t happening. The explosion in government isn’t happening. The public sector is shrinking.

The precise balance between public and private employment is tricky. But I think it is good to recognize what the actual direction of change is.

Lottery based admissions

From the Harvard Crimson:

For one thing, “excellence” in the Harvard admissions process—and at Harvard—has a lot less to do with virtuous character traits than with an ability to game the system. By placing a premium on students who go above and beyond in extracurricular realms, Harvard has attracted a number of truly incredible people but has also encouraged a high school arms race wherein kids cram their schedules with activities in an attempt to attract admissions officers.

By selecting for this kind of behavior, the admissions process doesn’t encourage real excellence, but, to use the novelist Walter Kirn’s term from his hilarious book and essay “Lost in the Meritocracy,” “aptitude for showing aptitude.” This may well be of use in students’ careers after college, but it is orthogonal if not antithetical to the goals of a liberal arts education.

I think that these factors are also an issue when you evalaute education at any level. People compete hard to get into top schools -- the same skills that they use to "game the system" also tend to ensure decent post-Harvard outcomes.

This makes evaluating the actual contribution of the education at Harvard, itself, tricky (to say the least).

h/t: Felix Salmon

More on owning preperty to vote

Commenter David made the point:

This criticism always sounds a little hollow--when people talk about interpreting the constitution 'as the framers intended', I'm pretty sure they don't mean the parts that later amendments changed.

What is difficult is that it is pretty clear from his quote that Judson Phillips really does believe this; go look. Now it could be that this is a minority view and not very typical.

But I jumped on it because this comment seemed to open a lot of issues:

But one of those was you had to be a property owner. And that makes a lot of sense, because if you’re a property owner you actually have a vested stake in the community.

So what happens if you have a mortgage on a property? Do you own it? What about a lien? How do you easily keep track of who has successfully sold their house (and thus lost the right to vote). When does this count? What if you buy a house right before election day?

Furthermore, how much property is enough? Do we value it by space? What about by value? In a country as diverse as the United States, either Wyoming or Manhattan is going to be complicated.

How would this interact with eminent domain? If the state seizes your house are you disenfranchised? Could this interact in unexpected ways with the idea of equal protection?

If you own land in city X but live (by renting) in city Y, where do you vote? What if you rent your house but own a cottage? Or a garden plot?

Finally, I wonder if the act of owning land really leads to stability given how liquid houses are. It might become more of a marker of wealth; arguing that only the wealthy should vote is a dangerous position.

So I was mostly struck by how badly this proposal seemed to be thought out and how complicated it would be to implement.