Randomized Social Experiments

In a really fascinating post, Alex Tabarrok discusses controversy about a random clinical trial set within a housing program. It's a very interesting example (and his quote from E. E. Peacock, Jr. was priceless). But it does bring up a serious question: in many areas of social policy there is no realistic way to blind experiments and, similarly, the idea of informed consent is hard to implement in practice.

Consider this point as well: the only way you will be able to get rational actors to consent to an experiment on housing is if all of the options in the intervention are equal or better than the status quo. That means you can never experiment to see if the current plans are working, insofar as informed consent is required. Nor can one really argue that coercion is not present given that people enter these programs out of desperation.

I think the real difference is that we have two different sources of utility: the user of the program and the agency (via the taxpayer) who is implementing the program. So it becomes a complex problem because the status quo might benefit the user (marginally) and harm the taxpayer/state/agency (greatly) but you will never get users to consent to testing how they will do with fewer benefits (or at least the incentives are wrong). Compare this with drug trials: only the patient’s ability to benefit is typically considered (although cost effectiveness may matter later in the approval process).

But the alternative is simply to not know what the right answer is and to risk getting stuck at a very suboptimal policy point. And that seems to be the wrong answer as well.

"I am not now and have never been a constructionist"

(This post also appears at Education and Statistics.)

After my last post thought I should run this titular disclaimer. For those of you not up on the subject, here's a definition from the well-written Wikipedia entry on the subject:

Constructivist teaching is based on constructivist learning theory. This theoretical framework holds that learning always builds upon knowledge that a student already knows; this prior knowledge is called a schema. Because all learning is filtered through pre-existing schemata, constructivists suggest that learning is more effective when a student is actively engaged in the learning process rather than attempting to receive knowledge passively. A wide variety of methods claim to be based on constructivist learning theory. Most of these methods rely on some form of guided discovery where the teacher avoids most direct instruction and attempts to lead the student through questions and activities to discover, discuss, appreciate and verbalize the new knowledge.

Don't get me wrong. For the right topic, executed the right way with the right teacher and class, this can be a great, wonderful, spectacular and really good approach to education. Unfortunately, education reformers (particularly the current crop), are not good at conditional problems. They tend instead to fall into the new tool camp (you know the saying, "to a man with a new hammer, the whole world is a nail.").

Worse yet, (and I'm afraid there's no nice way to say this) many of the educational theorists don't have a firm grasp on the subjects they are working with. This is never more plain than in constructionist science classes that almost entirely eschew lectures and traditional reading assignments and instead have the students spend their time conducting paint-by-numbers experiments, recording the results and performing a few simple calculations.

To most laymen, that's what science is: stuff you do while wearing a lab coat. Most people don't associate science with forming hypotheses, designing experiments, analyzing results and writing papers and, based on my limited but first hand experience, many science educators don't give those things much thought either.

The shining exception to the those-who-can't-teach-teach-teachers rule is George Polya. Though best known as an educational theorist, Polya was a major Twentieth Century mathematician (among his other claims to fame, he coined the term "central limit theorem") so he certainly fell in the those-who-can camp.

But it it important to note that Polya advocated guided discovery specifically as a way of teaching the problem solving process. I suspect that when it came to simply acquiring information, he would have told his students to go home and read their textbooks.

"Reasons to teach what we teach"

I've got a long post up on Education and Statistics on the reasons for including a concept in the mathematics curriculum and how those reasons should affect what we teach and how we teach it.

Here's an excerpt:

Where a topic appears on this list affects the way it should be taught and tested. Memorizing algorithms is an entirely appropriate approach to problems that fall primarily under number one [Needed for daily life]. Take long division. We would like it if all our students understood the underlying concepts behind each step but we'll settle for all of them being able to get the right answer.

If, however, a problem falls primarily under four [helps develop transferable skills in reasoning, pattern-recognition and problem-solving skills], this same approach is disastrous. One of my favorite examples of this comes from a high school GT text that was supposed to develop logic skills. The lesson was built around those puzzles where you have to reason out which traits go with which person (the man in the red house owns a dog, drives a Lincoln and smokes Camels -- back when people in puzzles smoked). These puzzles require some surprisingly advanced problem solving techniques but they really can be enjoyable, as demonstrated by the millions of people who have done them just for fun. (as an added bonus, problems very similar to this frequently appear on the SAT.)

The trick to doing these puzzles is figuring out an effective way of diagramming the conditions and, of course, this ability (graphically depicting information) is absolutely vital for most high level problem solving. Even though the problem itself was trivial, the skill required to find the right approach to solve it was readily transferable to any number of high value areas. The key to teaching this type of lesson is to provide as little guidance as possible while still keeping the frustration level manageable (one way to do this is to let the students work in groups or do the problem as a class, limiting the teacher's participation to leading questions and vague hints).

What you don't want to do is spell everything out and that was, unfortunately, the exact approach the book took. It presented the students with a step-by-step guide to solving this specific kind of logic problem, even providing out the ready-to-fill-in chart. It was like taking the students to the gym then lifting the weights for them.

This is a bit of a work in progress so if you have any relevant experience in mathematics education (and, yes, experience as a student definitely counts), I would greatly appreciate it if you went by and joined the conversation.

Interesting Passage

Via Tyler Cowen of Marginal Revolution fame comes an interesting article on school choice. I was particularly struck by this passage:

This approach helps explain why choice advocates — inclined to approach choice-based reform not as a regulatory question, but as one of justice and rights — have spent so much less time considering the dynamics of deregulation than have pro-market reformers in sectors like transportation, telecommunications, and cable television. Because education reformers have approached choice not as a matter of political economy but as a moral crusade, they have favored grand, sweeping claims over empirical reality.

I think that this has a lot of explanatory power for the postive treatment that charter schools are getting in the press and among pundits. Mark has written about this extensively on this blog.

I think breaking apart the moral and the policy dimensions would be a positive development. The issues with under-servicing of some groups are broad but there is a risk that alternatives could make things worse.

Homework and treatment levels

Previously on OE, we've talked about how norming, placebo and volunteer effects can cause methods like lottery-based analyses to miss significant sampling effects. Here's one more area of potential concern.

The basic problem here is familiar to clinical researchers. You're trying to determine the optimal level of a drug or therapy or exercise regime or type of food. There is an ideal dosage, the level of treatment you would prescribe if your only concern was optimizing the effect. Then there is what we might call a realistic dosage, one that takes into account factors like side effects and compliance. These levels vary from group to group which means the process of finding the right dosage is sensitive to sampling issues. The appropriate level is likely to be different for a group of college students and a group of senior citizens.

Teachers assigning homework face essentially the same problems. Assuming that goal is to improve students' score on a standardized test, there is an ideal homework assignment for each lesson, a subset of the problems available in the text that will tend to maximize the average score of a class (to keep things simple we'll limit the discussion, with no loss of generality, to homework assignments derived from the textbook).

The ideal assignment assumes that students have unlimited time and will complete all the assigned problems. The realistic assignment would take into account factors like time constraints, demands from other classes, compliance, burnout and parental pushback (I can tell you from experience that parents complain if they feel their children are being given excessive work and I would argue this is a good thing -- teachers should remember that their students' time also has value). The realistic assignment would optimize the class's average score when these constraints are in place.

Like the previously discussed norming, placebo and volunteer effects, a sampling/treatment level interaction with the homework assignment here can interfere with the ability of lottery-based analysis to detect sampling effects.

To see how this would work, consider a charter school like one in the KIPP system that very publicly acknowledges that students will be expected to do large amounts of homework each night (some schools even require parents to sign contracts agreeing to sign off on the students' homework). Students who apply for this school are aware of these requirements, as are their parents. This certainly raises the possibility that the optimal realistic homework level might be higher for these students, particularly if this is one of those schools that aggressively culls out the students who can't handle the workload.

If this is the case and if both the charter school and the public schools assign the optimal levels of homework for every class, you will have a sample based effect that will completely evade detection by a lottery-based analysis. That analysis will compare the performance of those who applied for the charter and were accepted against those who applied but lost the lottery. Since the rejected students will receive a treatment level that was optimized for the general population while the accepted students will receive a treatment level optimized to their particular subgroup, we expect the charter school students to do better, leading the lottery-based analysis to incorrectly conclude that there is no selection effect.

I used homework in this post to keep things simple but the principle applies just as well to most elements of the educational models of many highly praised charter schools -- longer days, Saturday instruction, extended school years, holding students responsible for more material on tests. Put bluntly, these schools get their results by giving students lots of work. Because of these results, these schools are often held up as examples for the rest of the educational system, but when you start with a self-selected sample then counsel out the students in that sample who have trouble keeping up, you should at least consider the possibility that the optimal workload for your student body is higher than the optimal workload for the general populace.

This doesn't mean that all high performing charter schools are simply running on undetected selection effects and it certainly doesn't mean that charter schools with high standards should lower them. What is it does mean is that selection effects in education show up in subtle and complex ways. It remains extraordinarily difficult to resolve these issues through observational means. Any study that claims to have settled them should be approached with great caution.

Arnold Kling

Arnold Kling is a skeptic about the importance of peer effects:

Back when the SAT was just math and verbal, I described us as living on the other side of a 250-point SAT gradient--in the better high schools, the SAT's were 1200, and in our high school they were 950. (I took the view that my own kids did not need high-achieving peers in order to do well on their own.)

I am not sure whether this is persuasive (although it does match my personal experience). But it is true that this is the sort of behavior that I would expect from peer-effects skeptics (and if we saw more of it there would be a much smaller premium on good schools for housing costs.

Thomas Jefferson on income inequality

From a letter to James Madison (courtesy of Brad DeLong):

I am conscious that an equal division of property is impracticable, but the consequences of this enormous inequality producing so much misery to the bulk of mankind, legislators cannot invent too many devices for subdividing property, only taking care to let their subdivisions go hand in hand with the natural affections of the human mind. The descent of property of every kind therefore to all the children, or to all the brothers and sisters, or other relations in equal degree, is a politic measure and a practicable one. Another means of silently lessening the inequality of property is to exempt all from taxation below a certain point, and to tax the higher portions or property in geometrical progression as they rise.

Whenever there are in any country uncultivated lands and unemployed poor, it is clear that the laws of property have been so far extended as to violate natural right. The earth is given as a common stock for man to labor and live on. If for the encouragement of industry we allow it to be appropriated, we must take care that other employment be provided to those excluded from the appropriation. If we do not, the fundamental right to labor the earth returns to the unemployed. It is too soon yet in our country to say that every man who cannot find employment, but who can find uncultivated land, shall be at liberty to cultivate it, paying a moderate rent. But it is not too soon to provide by every possible means that as few as possible shall be without a little portion of land. The small landholders are the most precious part of a state.

Profundity adjacent

I'm not a fan of the BBC's the Changing World. The show has interesting topics but the episodes I've caught have an annoying habit of almost saying something profound and insightful then veering off at the last minute. This tendency to be profundity adjacent is particularly notable because, on KPCC, Changing World airs a couple of hours after This American Life, a show with an extraordinary ability to uncover the genuinely important aspects of a complicated story.

The criticism certainly holds for this episode on chess. Repeatedly, I found myself caught up in an interesting thread only to have it end right before it got to the good part. Still, they were interesting threads and if your interests include abstract strategy games and their role in fields like education, you should probably check this out.




http://www.thechangingworld.org/archives/2010/44.php

On Salmon's blog, even the comments are quotable

Felix Salmon has a good post on the economic impact of extending unemployment benefits but for me the most insightful observation was found in the comments:

What’s missing from this discussion is the part about optimization of deployment of resources. If a skilled worker, say a skilled aluminum welder, can stay on unemployment long enough to find a skilled aluminum welder job at higher pay and productivity, that’s good for GDP and has a higher multiplier. But if that worker has to settle for a security guard job because he or she must pay the rent, then there is a net loss of productivity, GDP, and related multipliers.

Wargaming

This article is a very interesting look at the rise and fall of wargaming. Of especial interest was:

In 1982, the 30,000 subscribers of Strategy & Tactics, SPI's flagship magazine, were the most avid wargame enthusiasts on the globe. More than 50% of them, per SPI's own feedback, owned 100 or more wargames; most of them bought a dozen or more games every year, not counting the games they received as subscribers to the magazine. SPI estimated that perhaps 250,000 people, in the whole of North America, had ever bought a wargame; and the 30,000 subscribers to S&T bought an enormously disproportionate number of the games sold.

They were the hard core, the fundament upon which the whole wargame industry was built.

So naturally, when TSR took over SPI, the first thing it did was give the finger to S&T's subscribers. The first thing it did was say to the best customers of its new subsidiary, "Go take a hike; we don't want your custom; your concerns mean nothing to us."

You see, almost every magazine in the country can declare bankruptcy tomorrow, if it wants to, because every magazine in the country has enormous liabilities: the obligation to provide issues to its subscribers. The subscription money came in long ago, and has been spent, and that liability remains. Magazines continue, and make money, because they sell advertising, and expect their subscribers to resubscribe. But SPI had no money -- and S&T had 30,000 subscribers. Over a thousand were life-time subscribers, owed issues in perpetuity in exchange for no further income.

TSR didn't want the liability. Fulfilling S&T's existing subscriptions would have cost it money.

So, TSR decided, it would not honor any subscriptions.

TSR had taken over SPI's assets, but not its liabilities, so they claimed; therefore, they had no obligation to S&T's subscribers.

Gosh. Guess what happened? Few of S&T's subscribers reupped. Few wanted to send more money to the company that had just ripped them off.

And few of them ever bought any of the wargames TSR began to publish.

And TSR never could figure out why their wargames never sold.

I think that this example is very interesting for a couple of reasons. One, the fact that this was such a large setback to the entire field makes it clear just how important barriers to entry can be in making competition difficult (it would have been infeasible to develop the network that had just been destroyed by another publisher). Two, is the market power of corporations. With their ability to raise equity, they can purchase viable business models but they do not necessarily manage them well.

For a luxury good, like wargames are , it is unfortunate for people who enjoyed them but hardly a tragedy -- other goods will have the chance to thrive in the market instead (and perhaps this is a form of creative destruction?).

But imagine this process for an essential good: like water, electricity or education. I think we should keep examples like this in mind when considering the benefits of privatization. There are massive upsides to capturing the efficiency of a functioning market -- but only if players are allowed to fail.

Funding Cuts

From Dead Dad:

Of course, that refers only to general college reserves. It’s also common for various programs to have reserves of their own, earmarked for specific purposes. The college foundation might have reserves dedicated for certain scholarship awards. Some grant-funded programs will have reserves for specific functions and only for those functions. (In the context of multiyear grants, for example, it’s common to have ‘carryover’ of excess funds from one fiscal year to the next. That’s frequently allowed, but that doesn’t give license to transfer the extra grant money to the general college budget.) In cases like those, money comes with strings attached, and violating the terms of the money involves forfeiting the money. You can’t just move it around.

This lack of flexibility actually highlights one of the difficult issues with cuts in grants. If grants are cut by 10%, you don't have the discretation to eliminate 10% of the ongoing projects to make sure that the rest are successful. In the same sense, going after indirects to try and make up these losses would lead to the defunding of other operations.

The modern university budget looks remarkably inflexible to me, which makes planning for adverse financial circumstances appear to be extremely tricky. Not only do you have to cut but what you can cut can be remarkably constrained. This can lead to very poor optics (where something that looks non-essential is fully funded whereas a core operation simply lacks funding).

It is a tough place to be in!

Education reform and the two kinds of knowledge

"Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information upon it. When we enquire into any subject, the first thing we have to do is to know what books have treated of it. This leads us to look at catalogues, and at the backs of books in libraries."
— Samuel Johnson

This came up while I was working on a longer piece but I decided it deserved a short post of its own.

In the two and a half centuries since Dr. Johnson's time, the amount and the value of the second kind of knowledge has increased substantially (as it had in the preceding centuries). Cheap printing, the internet and telecommunications (consulting an expert qualifies as the second kind) have given us ready access to a stunning array of information.

With a few exceptions, the ability to find and manipulate information is far more important than the ability to simply repeat it, but the push for accountability in education often focuses strictly on the first kind.

update: You can find a related discussion here.

Someone finally notices the orphan technology

As I mentioned before, over-the-air broadcasting has been a conspicuous hole in any number of media stories, probably due to the demographics of that market and the lack of interested parties pushing the story. Eventually though, once the market included enough young, photogenic professionals, the New York Times did pick up on the story (from this Sunday):

Many pay TV customers are making the same decision. From April to September, cable and satellite companies had a net loss of about 330,000 customers. Craig Moffett, a longtime cable analyst with Sanford C. Bernstein, said the consensus of the industry executives he had talked to was that most of these so-called cord-cutters were turning to over-the-air TV. “It looks like they’re leaving for the antenna,” he said.

The article has its problems. There are factually challenged passages like this:

The new antennas do pull in more programs than your grandfather’s rabbit ears, because of new channels that broadcasters added during the transition to digital signals. The broadcasters can fit multiple digital channels into the same frequencies that used to carry one analog channel.

In St. Paul, for example, where Ms. Bayerl lives, there are extra channels from ABC and NBC with local news and weather, four public television channels and a music video channel. Big markets like Los Angeles have 40 or more channels, according to Nielsen.

(I live in LA county and, according to my digital converter I get 114 channels. There are at least a half dozen more I could get if I wanted to invest in a better antenna set-up. When you're off by a factor of three, "or more" just doesn't cut it.)

The article also completely overlooks the role that the Hispanic market plays in the over-the-air broadcasting story which is a bit like ignoring the role commuters play in the NPR audience.

Still, compared to what we've seen up till now, this account is almost cutting edge.

Update:

While picking up a printer cartridge this morning I fact-checked the following passage from the NYT story:

Modern antennas, which cost $25 to $150, pick up high-definition signals that can actually be crisper than the cable or satellite version of the same program, because the pay TV companies compress the video data.

Actually HDTV compatible antennas the one I use for my 114 channels are available at Target start at $10.98.

NIH news

From Medical Writing, Editing & Grantsmanship:

According to the AAMC, seems most likely these CRs will be the high-point of FY11:

During debate on the measure, Rep. Jerry Lewis (Calif.), the ranking Republican on the House Appropriations Committee, again expressed opposition to “any potential omnibus spending bill the Democratic leadership may be planning to bring to the House floor before the end of the year.” He also opposed extending the CR for the balance of FY 2011 at current level, which he described as “frankly, too darn high.”

I won't argue the merits or deficits of this view at the policy level except to say that, from a personal point of view, this news doesn't help my plans to start a career in this area.

Perils of studying Small Effects

From Andrew Gelman:

For example, suppose you're a sociologist interested in studying sex ratios. A quick review of the literature will tell you that the differences in %girl births, comparing race of mother, age of mother, birth order, etc, are less than 1%. So if you want to study, say, the correlation between parental beauty and sex ratio, you're gonna be expecting very small effects which you'll need very large sample sizes to find. Statistical significance has nothing to do with it: Unless you have huge samples and good measurements, you can pretty much forget about it, whether your p-value is 0.02 or 0.05 or 0.01.

I think that this is correct and ties into a larger narrative: there are a lot of small effects of great theoretical interest that are really hard to show in actual data due to the limitations of realistic sample sizes. Observational epidemiologists often try to handle this by looking at prescription claims databases (for example). But these studies create a new problem: there is a serious concern of unmeasured confounding due to factors that are not measured in these data sources (e.g. smoking, alcohol use, diet, exercise). It's not really a big step forward to replicate lack of power with potential bias due to confounding.

I think the real issue is simply that small effects are difficult to study, no matter how interesting that they are. So I think that Andrew Gelman is right to call for the interpretation of these studies to done in the context of the larger science.