Implications of "counseling out" part I -- metrics

This comment from Michael Bish raises a couple of big questions:

One important point is that there is a big difference between counseling a student out because they have low test scores and counseling them out because they are actively disrupting the education of their peers.

The first issue here involves metrics. The idea that you should build a school system around a handful of test scores is one of the central tenets of the reform movement. Test scores are supposed to drive funding, contracts, allocation of resources, evaluations, bonuses, even terminations, but if you give certain schools more freedom to get rid of disruptive students the whole system breaks down.

Disruptive students not only take up time from the teacher that could be used in instruction; they also make it difficult for students to concentrate and tempt other others to act up. The result is that everyone's test scores go down.

By getting this student transferred you've not only raised the scores of an entire classroom of your students; you've lowered the scores of a comparable number of students in a public school in the same area. Since your school's evaluation, funding, and future contracts are dependent on your performance relative to other schools in the area, you can get a substantial double lift out of that single transfer.

And since public school to a large extent have to work with the students they are given, charter schools always win this one.

"No, I said 'self-select for attitude.'"

One of the many weird aspects of the charter school debate has been the non-exchanges over selection effects. Critics suggest that students who choose to go through the application process will tend to be more motivated, more committed to school, more interested in college, and more open to the idea of a heavier work load and will tend to have more stable, supportive families.

Here's an example:

Some observers, such as the authors of The Charter School Dust-Up, say that KIPP's admission process self-screens for students who are both motivated and compliant, from similarly motivated and compliant—and supportive—families. Parents must commit to a required level of involvement, which rules out badly dysfunctional families.

This would seem to be fairly straightforward. Almost every school could do a good job if it were populated solely by students who worked hard, followed instructions and had the full support of their families. A strong selection effect here could explain away most or all of the positive results observed in KIPP schools. This hypothesis even raises the possibility that KIPP schools are actually doing worse than public schools would under comparable circumstances.

(You'll notice that no mention is made of race, poverty or test scores.)

Now here's how SRI responds:

KIPP schools’ higher-than-expected test score results draw both attention and claims that they “cherry-pick” high-achieving students from poor neighborhoods. This is the first report to closely scrutinize the praise and criticisms associated with KIPP, as well as key challenges facing Bay Area KIPP schools today.

In the three KIPP schools where they were able to draw comparisons, SRI researchers found that students with lower prior achievement on the CST were more likely to choose KIPP than higher-performing students from the same neighborhood, suggesting that, at least at these schools, cherry-picking does not occur.

And here's the much-touted Mathematica report:

Our nonexperimental methods account for the pre-KIPP (baseline) characteristics of students who subsequently enter KIPP schools: not only demographic characteristics such as race/ethnicity, gender, poverty status, and special education status, but also their achievement test results for two years prior to entering KIPP. We examine the achievement levels of the students for up to four years after entering KIPP schools. In our preferred models, we compare these trajectories to the achievement trajectories of a matched set of local public school students who have similar achievement test results and demographic characteristics in the baseline period (typically third and fourth grades) but who do not enroll in KIPP.

The heroin's still doing the heavy lifting -- why Ivy League legacies work

From Christopher Shea's Boston Globe column:

Richard D. Kahlenberg, editor of the forthcoming book "Affirmative Action for the Rich: Legacy Preferences in College Admissions," points out that universities in other countries don't give so-called legacy preferences to sons and daughters of their alumni. (Even Oxbridge colleges don't, despite the class-bound history of British education.) So, he asks, why on earth do we do it in America?

Broadly speaking, students go to college in search of four things: certification; instruction; reputation; and connections.

In terms of certification, any well-accredited school would do. In terms of undergraduate instruction, the best deal for the money (and perhaps the best deal period) is the small four-year school. (I'm leaving this as an assertion but I'm fairly confident I can argue the point if anyone wants to debate.)

In the next two categories, however, the Ivy League cannot be surpassed, in part because of the legacy system.

Without loss of generality, look at Harvard. The student population of the school consists entirely of two overlapping groups: people who can get into Harvard; people whose parents can get them into Harvard.

The first group is hard-working, ambitious and academically gifted. Assuming the number of need-based legacies is trivial, the second group comes from families that are wealthy (they're paying for a Harvard education) and well-connected (at least one parent went to Harvard).

Putting aside luck, you can put the drivers of success into three general categories: attitude, drive and work habits; talent, intelligence and creativity; reputation and connections. It is possible to succeed with just one of these (hell, I can think of people who made it with none), but there is a strong synergistic effect. A moderate talent who works hard and has connections will generally go farther than a spectacular talent who's lazy and isolated.

Connections are governed by the laws of graph theory. I'm not going to delve too deeply into the subject (since that would require research and possibly actual work on my part), but as anyone who has read even the cover blurbs on Linked or Small Worlds can tell you, adding a few highly connected nodes (let's call them senator's sons) can greatly increase the connectivity of a system.

It would be interesting to model the trade off between picking a well connected legacy over a smarter, harder-working applicant given the objective of producing the greatest aggregate success. Because of the network properties mentioned above, it wouldn't be surprising if the optimal number of legacies turned out to be the 10% to 15% we generally see.

Optimized or not, this mixture is almost guaranteed to churn out fantastically successful graduates regardless of what the schools do after the students are admitted. I'm certain the quality of instruction on the Ivy League schools is very good, but, like most education success stories, the secret here is mostly selection and peer effects.

Update: For a different interpretation (this time with actual data), check out this post at Gene Expression.

Updated update: Why doesn't spell check work in the title field?