Monday, August 30, 2010

Hazards of sweeping generalizations

Commenter Nat makes a good point about sensitivity versus specificity:

The other reason for having 100% sensitive tests at the cost of specificity is because of the clinical tradeoffs that occur because you have done that test.

If, for example, the treatment subsequent to a test is vitamin supplementation which should have next to zero complications then 100% sensitivity is the face of nasty complications caused by non-treatment makes quite a lot of sense.


In some of the areas that I work, like pain, we are denied these elegant trade-offs. However, I also do work in coagulation and there are good examples of this type of trade-off there. For example, despite the limited evidence of clinical utility, it can make sense for people with Homocysteine and MTHFR mutations to take b-vitamins. Similarly, a few false positives have very limited impact on the patients involved as the risk of taking a b-vitamin supplement (in the first world where economic hardship is unlikely) is small.

So this is a good reminder that there are no sweeping generalizations in epidemiology.

Saturday, August 28, 2010

Megan's List

Blogger Megan McArdle has a list of things where her best guesses turned out to be incorrect in the face of evidence. We all have these cases and it is good to look back and see where foresight failed us. There is nothing like data to help us revise our internal prediction algorithms.

This point, though was chilling:

I believed that over reasonably long time-frames, modest investments in equities would allow you to retire in comfort.


I am mildly interested (as a hobbyist) in personal investment. But I think the implications of this are far broader than one sentence really captures. It makes the whole idea of shifting Social Security (as an insurance program against poverty in old age) into personal accounts somehow less appealing. It also says very interesting things about the rate of retirement for those of us who started our careers late (due to mid-career changes). The more I think about this issue, the more it has very profound implications for the way our work force will evolve.

But, sadly, I also think that this point is the best reading of the current stagnation of equities, even if the long run is better it is hard to have the level of confidence that I once did.

Friday, August 27, 2010

Trade-offs between Type I and Type II error

I was reading this blog post by Andrew Gelman on a test that is 100% accurate for alzheimer's disease. Following the initial post up, it appears that the test has only 64% specificity.

But the feature of this discussion that I find the most interesting is that the decision to choose between sensitivity and specificity is a judgment that people seem to be very poor at. Consider pain medication. If you want to make sure that everyone in serious pain gets appropriate pain control than some fraudsters will get illicit narcotics. Alternatively, if you make the screen tight that nobody is able to obtain narcotics via "fake pain" then some real cases will be undertreated.

We see the same thing with releasing people from prison. Even if former prisoners only committed crimes at the rate of the general population, at least some crimes could be prevented by a tougher release policy. Of course, this line of reasoning leads to absurd conclusions -- we could completely eliminate adult crime by jailing everyone for life on their eighteenth birthday. We see the same thing in the Ray Fisman argument for getting rid of 80% of teachers during probation – it is so important not to make a mistake and keep an inferior teacher that we should fail to hire many good teachers just to make sure we have no sub-standard ones.

But people don't seem to like to make these trade-offs. In the case of the test for Alzheimer's disease, the authors could have been a lot more specific if they were willing to give up sensitivity. But, for some reason, people seem to prefer to end up at one extreme of a scale rather than the middle (where the value of the test is maximized).

It's a phenomenon that I wish I understood better.

Thursday, August 26, 2010

Badly needed break from Fisman

With two brilliant clips from the Daily Show


The Daily Show With Jon StewartMon - Thurs 11p / 10c
Extremist Makeover - Homeland Edition
www.thedailyshow.com
Daily Show Full EpisodesPolitical HumorTea Party


The Daily Show With Jon StewartMon - Thurs 11p / 10c
The Parent Company Trap
www.thedailyshow.com
Daily Show Full EpisodesPolitical HumorTea Party

US Mobility

In a ncie article, the mobility myth, it is pointed out that only 2.7% of Americans cange states each year. My experience is different than that but that is likely because I am in Academia. Still, I am definitely aware of the risks that changing states brings (as you can never know if a state will work out or not in advance).

On the research side, however, this fact is good news for database research as the "lost to migration" rate is low enough to make it unlikely that we will get serious bias in state level medical claims data. That is really useful to know when evaluating MedicAid studies.

Genetic Epidemiology

John Cook has a post on predicting height using genes. He quotes:

A 2009 study came up with a technique for predicting the height of a person based on looking at the 54 genes found to be correlated with height in 5,748 people — and discovered the results were one-tenth as accurate as the 125–year-old technique of averaging the heights of both parents and adjusting for sex.


I suspect that this issue is the central one facing genetic epidemiology. While it is possible that the approach of averaging the height of the parents includes some environmental information, it is a pretty strong comment on the predictive power of genes if that is the actual answer.

More likely, I think, is the idea that complex and important characteristics are due to many, many genes (all of which have a modest influence). The makes sense from a selection point of view (characteristics like height need to be stable) but makes the project of prediction using genes extraordinarily complicated. I don't know if there is a simple answer or not but it definitely provides some challenges for the paradigm of the classic epidemiological study.

Wednesday, August 25, 2010

Mystery (Education Question)

On thing that amazes me in the education debate is that people of all political stripes seem to agree that education is in a crisis. Consider Jonathan Chait (whom I think it is clear is a liberal) who seems to agree that teacher firings make sense. Yet, as Mark notes, America leads in elementary education.

So why are the two so often conflated?

It could be the "big lie" where a falsehood is said so often that the other side starts to believe it. But people are usually more sophisticated than that.

Another possibility is that we have lost perspective on the alternatives. We worry about reluctance to fire teachers but forget that private alternatives are not inexpensive. From Marginal Revolution:

A New York City charter school set to open in 2009 in Washington Heights will test one of the most fundamental questions in education: Whether significantly higher pay for teachers is the key to improving schools.

The school, which will run from fifth to eighth grades, is promising to pay teachers $125,000, plus a potential bonus based on schoolwide performance. That is nearly twice as much as the average New York City public school teacher earns, roughly two and a half times the national average teacher salary and higher than the base salary of all but the most senior teachers in the most generous districts nationwide.


However, this still doesn't explain the odd consensus of left and right as it seems improbable that many people are fighting for a serious increase in education costs.

Most likely, I suspect, the the current American focus on short term results. When we do annual ratings of employees, we do not consider issues of long term dedication -- we know people are simply going to move on anyway. Consider this report:

Among jobs started by workers when they were ages 38 to 42, 31 percent ended in less than a year, and 65 percent ended in fewer than 5 years.


Is it possible that, with 65% of middle aged workers holding a job for less than 5 years, that we have simply lost the sense of how to build long term loyalty and dedication?

The Old Shell Game -- Why you have to keep your eye on Ray Fisman (and no, we're not quite through with the second paragraph)

Ray Fisman's Slate article "Clean Out Your Desk" is something of a greatest hits of the education reform movement, covering most of the standard arguments and carefully selected statistics that are brought up again and again by the advocates. It is, therefore, fitting that FIsman gets no further than the second paragraph before getting to this:
New York City Schools Chancellor Joel Klein often quotes the commission before discussing how U.S. schools have fared since it issued its report. Despite nearly doubling per capita spending on education over the past few decades, American 15-year olds fared dismally in standardized math tests given in 2000, placing 18th out of 27 member countries in the Organization for Economic Co-operation and Development. Six years later, the U.S. had slipped to 25th out of 30. If we've been fighting against mediocrity in education since 1983, it's been a losing battle.*
The OECD tests are the book of Revelations of the education reform movement, the great ominous portent to be invoked in the presence of critics and non-believers. Putting aside questions of the validity and utility of this test (perhaps for another post if my stamina holds out), we would certainly like to be in the top ten rather than the bottom.

But before we concede this one, lets pull out our well-thumbed copy of Huff and take one more look. Whenever one side in a complex debate keeps pulling out one particular statistic, you should always take a moment and check for cherry-picking.

Is Fisman distorting the data by being overly selective when picking statistics to bolster his case? Yes, and he's doing it in an egregious way.

Take a look at at the Trends in International Mathematics and Science Study. Here's a passage from the executive summary from the National Center for Education Statistics:

In 2007, the average mathematics scores of both U.S. fourth-graders (529) and eighth-graders (508) were higher than the TIMSS scale average (500 at both grades). The average U.S. fourth-grade mathematics score was higher than those of students in 23 of the 35 other countries, lower than those in 8 countries (all located in Asia or Europe), and not measurably different from those in the remaining 4 countries. At eighth grade, the average U.S. mathematics score was higher than those of students in 37 of the 47 other countries, lower than those in 5 countries (all of them located in Asia), and not measurably different from those in the other 5 countries.
We could spend some time in the statistical weeds and talk about the methodology of TIMSS vs. OECD's PISA. TIMSS is the better established and arguably better credentialed, but both are serious efforts mounted by major international organizations and it would be difficult to justify leaving either out of the discussion.

If Fisman had limited his focus to the education of high school students and simply ignored the data involving earlier grades, we would have ordinary misdemeanor-level cherry-picking. Not the most ethical of behavior, but the sort of thing most of us do from time to time.

But Fisman does something far more dishonest; he quietly shifts the subject to teachers in general and often to elementary teachers in particular (take a good look at the study that's at the center of Fisman's article).

This means that, when you strip away the obfuscation, you get the following argument.

1. The best metrics for tracking American education are international rankings on math tests;

2. The best way of improving America's education system is fire massive numbers of teachers, including those in areas where we are doing well on international rankings on math tests.

The bad news here is that we have a long way to go to make it through Fisman's article and it doesn't get much better, but the good news is that we're through with the second paragraph.




* Fought, for the most part with Klein and Fisman's battle plan but we've already covered that.

More on Avandia

Here is an interesting post on Avandia. It points out the discrepancy between the number need to treat (to show benefits on the key endpoints) and the number needed to harm (with a myocardial endpoint). Now, the author is neglecting the uncertainty in these two numbers.

But it's a pretty clear that if the number needed to harm is 50 and the number needed to treat to prevent a serious outcome is 1000 then the medication is likely not favorable on the cost-benefit analysis.

There are cases where the risk-benefit calculation is a subtle problem and it is always tricky to withdraw a drug that showed actual benefits in the original clinical trials. But it is looking increasingly like Avandia may carry more risks than benefits making it an exception to the rule.

Monday, August 23, 2010

Starting from the beginning -- Ray Fisman's sins of omission

The centerpiece of Fisman's recent Slate article, "Clean Out Your Desk," is a deeply flawed analysis proposing that four out of five probationary teachers should be fired, but the problems with the article aren't limited to that one piece of research; they permeate the article, starting with the very first two paragraphs:

In 1983, a presidential commission issued the landmark report "A Nation at Risk: The Imperative for Educational Reform." The report warned that despite an increase in spending, the public education system was at risk of failure "If an unfriendly foreign power had attempted to impose on America the mediocre educational performance that exists today," the report declared, "we might well have viewed it as an act of war."

New York City Schools Chancellor Joel Klein often quotes the commission before discussing how U.S. schools have fared since it issued its report. Despite nearly doubling per capita spending on education over the past few decades, American 15-year olds fared dismally in standardized math tests given in 2000, placing 18th out of 27 member countries in the Organization for Economic Co-operation and Development. Six years later, the U.S. had slipped to 25th out of 30. If we've been fighting against mediocrity in education since 1983, it's been a losing battle.

Notice that strange gap of more than a quarter century? Other than mentioning increased spending per capita* and citing a couple of ominous sounding statistics, Fisman doesn't say a word about what happened since. There is no mention of what the response was to "A Nation at Risk." You could easily come away with the impression that there was no response, that educators had simply gone on with business as usual.

This is a common rhetorical trick in the educational reform movement.: to point out various facts suggesting a dangerous decline over the past two or three decades then quickly change the subject (sometimes citing "A Nation at Risk" to add a note of the Cassandra Syndrome). If only we had done something then, we wouldn't be on the precipice now.

The primary flaw in this narrative is that there was a response to the report, it was swift and sweeping, and mostly it consisted of the types of change reformers like Fisman, Klein, and Ben Wildavsky continue to push for to this day: importing techniques and philosophies from the private sector; encouraging privatization and entrepreneurs; basing the evaluation of schools on objective metrics (particularly standardized tests of student performance).

By the late Eighties, when I went into teaching, it was difficult to find a school without a mission statement. Staff development by then consisted almost entirely of the kind of training/motivation seminars that I would encounter a few years later working for Fortune 500 companies. Business jargon was all the rage. My first encounter with the school of education was when the dean gave us a talk on how Tom Peters and In Search of Excellence were going to revolutionize education.

For a while, I taught in a high school where the principal was known to say that he didn't like to base teacher renewal or promotion decisions on standardized test scores. He was careful not to say he wouldn't. That would have been a blatant lie. We knew he would make our lives miserable if we didn't teach to the test. He knew we knew it. But he maintained at least a vaneer of plausible deniability.

One particularly spineless history teacher spent about a month doing nothing but drilling facts that were likely to be on the test. No discussions. No writing assignments. No additional reading. No attempt to put the material into any kind of meaningful context. But his scores were good.

By the early Nineties, within less than a decade of the "Nation at Risk" report, states were starting to pass charter school laws. Pushes for merit pay and weakening tenure intensified. Faith in business as a source of answers for schools continued.

Education reform has proceeded in more or less a straight line for more than a quarter century without much that can be held up as a clear success. This isn't necessarily a damning criticism. Reformers like Fisman and Klein could honestly argue that the current state of education is mixed, not as good as it could be but not as bad as it would have been if steps had not been taken, or they could argue we have a classic case of half measures, that these reforms would solve our problems if fully implemented but in their watered down form they can do no good.

Both of these arguments are honest and defensible.

What they can't honestly do is imply that we are where we are because we didn't listen to them.




*Why per capita and not per student?

Thursday, August 19, 2010

Ray Fisman and the Tierney Ratio

The Tierney Ratio (sometimes called the Tierney Test because people love alliteration) is a measure of journalistic mediocrity named for its frequent subject, John Tierney. You find the Tierney Ratio of an article by counting the number of words it takes to address all of the significant problems in the article, then dividing that by the article's word count.

As you might expect, Tierney Ratios vary greatly from author to author. The sorely-missed Olivia Judson maintained a TR of virtually zero while writing for the New York Times while John Tierney, a science writer with no appreciable background in or aptitude for science, routinely had observed TRs in excess of one or two. (it is possible that Judson was kept in the Op-Ed rather than Science section out of concern that she would unfairly lower the latter's average.)

The value of the Tierney Ratio is somewhat limited by its serious data censoring problem (analogous to this well-known example). Faced with articles and essays of sufficiently low quality, researchers are almost always forced to leave significant mistakes, distortions and fallacies unaddressed.

Which brings us back to Ray Fisman's recent column in Slate, which reaches an almost Hellmanesque level of inaccuracy. Getting a true TR on something like this is an extraordinarily tedious job so the readers who aren't into hardcore education wonkery might want to skip the next few posts. You'll know it's safe to come back when we start posting Daily Show clips again.

Data censoring and Tootsie Pops

There is reason to suspect undercounting.

Wednesday, August 18, 2010

Desperately Seeking Suzanne (Null)

In Life in Hell, Matt Groening once asked if there was anything scarier than an open-mic poetry night. As a general rule, I have the same reaction to comment sections. There are exceptions (Andrew Gelman's site come to mind) but most of the time your chances of happening upon an intelligent and insightful conversation are better when you sit down between two strange drunks in an unfamiliar bar.

So you can understand why I initially skipped over the comments to Ray Fisman's recent post (if comments are usually less intelligent and well-written than the articles they accompany, just imagine the Stygian depths these would have to sink to in order to follow Fisman). Fortunately Joseph did brave the bottom of the web page and discovered that the comments here were actually better than the piece that inspired them.

The best of that very good crop were the entries was by Suzanne Null, who is (I believe) an education professor in the Northeast. [update, strike that last part. It looks like Suzanne is a fellow Westerner.] In this series of comments she takes down Fisman brick by brick:
Didn't Fisman's teachers ever teach him to conduct some research and check the validity of his sources (there is better and more recent information than the 1997 research he cited) before he publishes something? Virtually all of the information in this article has already been debunked. See Ravitch's copiously-researcherd "The Death and Life of the Great American School System" (particularly Chapter 9) and practically everything by Stanford's Linda Darling-Hammond. For example:

1) There is a great deal of evidence that better training helps teachers improve instruction (see research by Darling-Hammond and on the "Research" section of the www.nwp.org site). Teachers are "made" (not born) through training, professionally supportive school environments, and supportive communities. Experience makes a difference in teacher effectiveness (Ravitch 190) and one of the most major problems with the teaching profession is its high rate of attrition; many teachers leave the profession by their fifth year.

2) Despite what this article says about identifying "bad" teachers, we haven't yet found a reliable way to identify who the "bad" teachers are. Test scores are one-dimensional and subject to numerous validity and reliability issues (Ravitch 152-154). In addition, despite the claims made in this article, test scores can vary significantly by teacher from year to year because there is so much variation among the students in the teachers' classes. (Ravitch 185-186). A teacher who the tests identify as "high performing" one year might appear to be "low performing" the next.


3) The article insinuated that schools can "close the gap" simply by hiring the top quintile of teachers. This research comes from Gordon, Kane, & Stagler 2006; Hanushek & Rivkin (2004), and Sanders (2000), all cited by Ravitch (183-184). This has also been debunked because the learning gains cited in these articles don't persist over time (Jacob, Lefgren, & Simms 2008) and because of the general unreliability of the tests, particularly when used for the purpose of evaluating teachers, which was not the primary goals when most of these tests were designed.

4) The effectiveness of Teach for America (TFA) has been inconclusive (Rativich 188-191). For example, an extensive study by Darling-Hammond's found that TFA teachers "had a negative or non-significant effect on student achievement" (2005, cited by Ravitch 189). Thus "degrees from prestigious colleges" are also NOT a predictor of effective teaching. In any case, it is delusional to believe that the entire country can sustain the constant turnover of teachers that has characterized TFA (particularly given schools' current budgets for teacher pay) or that this level of turnover would be desirable for our students (Ravitch 190).

5) The research on the "cumulative effects" of attending NYC charter schools has been proven to be invalid. Charter schools are not all successful -- some post higher test scores than their comparative "public" schools and others post lower scores. When they have higher scores, it is usually because they take the students who chose to enroll or enter the lottery system. These students and their families tend to be more engaged with their education in general, and thus tend to perform better, no matter what the teacher does. Charter schools also take fewer students with special needs, such as Special Education students or second-language learners. When researchers have adjusted for these differences in their data pools, they have found no significant differences between charter schools and public schools (Ravitch 140 -143).

...

What's particularly worrisome and insidious about the author's arguments are that they will further harm students within our school system. If the "thought experiment" of abandoning teacher selection based on qualifications and teacher training is ever carried out in favor of allowing anyone to try to teach so that the "data" can winnow out the top 20%, it will mean that our students will bear the brunt of training and selecting teachers. They will be subjected to a revolving door of completely untrained teachers, and they will lose educational time and opportunities as they experience the steep learning curve that is present for teachers in their first two years. Our students deserve trained and experienced teachers; they don't deserve to be the guinea pigs that have to test out anyone who walks in off the street.
...

If we really want to overcome "mediocrity" in schools, we should focus on retaining the best teachers, giving them the professional freedom and support to do their jobs, and incentives for high performance, not just on tests, but on other measures of teacher success. Since I began teaching in 2000, I've watched many of the hardest working, most committed, and most motivated teachers leave. Those who stay are a few of the truly exceptional ones, and many of the ones who are happy to administer lectures and scan-tron tests and then go home. Teachers have few opportunities for professional advancement (aside from maybe becoming a principal), and few incentives to go "over and above" in their jobs. If we really want to improve education, our school SYSTEMS' tendency to support the mediocre and discourage (or even fire) the best is what will need to change. This change will require better working conditions, better support, more resources, smaller classes, and even better pay incentives for our hardest working and best-performing teachers.
...

I would add that the whole "martyr" or indolent "loser" dichotomy presented in the media's portrayals of teachers allows our society to evade responsibility for actually improving schools. If the best teachers are great because they CARE so much about their students and are willing to sacrifice so much, and if as the article says they are "born great," then they won't require smaller classes, better materials, more manageable work responsibilities, or higher pay. And if the worst teachers are indolent, then more money isn't going to help them anyway. The entire construction allows our culture to continue to alternately lionize and blame teachers while doing nothing that would actually help support teachers in their endeavors to help students learn.
...

Actually, the one strategy that's been proven to raise test scores is to winnow out the low-scoring students. This can be accomplished by re-drawing school attendance boundaries, creating "choice" or charter schools (which of course don't have the "resources" for Special Education students, second-language learners, or students with behavioral issues), or by "encouraging" the low-performing students to drop out or leave. The schools that have done this have been able to tout the "excellence" of their school management and teacher training, and their principals and superintendents have often gotten promotions and large pay raises.

So maybe our schools should all try that.
...
Just to clarify, this last suggestion was facetious. If our only way to "improve" our schools is to stop serving all of our students, is that a form of "success" that's worth having?
...

EB, I've particularly heard stories about nepotism and favortism from teachers in rural schools, so I know it happens. But many teachers' major fear about "performance based" pay is that it will be subject to the same dynamics. Even if teachers are evaluated solely on "data" such as test scores (which isn't a good idea for other reasons), it is very easy for principals to stack the deck against teachers they don't like by giving them the lower-performing students, making them change grades, or subjects, giving them unfavorable schedules, etc. I've already heard from some teachers I know in a rural area that principals will "drive a teacher out" by say, transferring them from fourth grade to first, only to then blame the teacher when test scores dip because the teacher hasn't had time to accumulate the practice and materials for the new grade level.

Many teachers are supportive of standards, accountability, and even incentive pay, but they want to be evaluated in fair and equitable ways.
Suzanne, if you've got a place you're posting on a regular basis, let us know and we'll add you to our blogroll. What you have to say deserves the widest possible audience.

Tuesday, August 17, 2010

A Partial N-space of Eduction Reform -- another preblogged footnote

If we are going to have an intelligent conversation about education (which at this point would be a refreshing change of pace), we have to start by thinking about the n-space. There are multiple dimensions that have to be considered here. As long as the debate fails to acknowledge them or approaches them in a sloppy way, the analyses will continue to be fatally flawed.

We could look at this on the level of classes or individual students, but in this case it probably makes the most sense to think of each school as representing a point in this multidimensional space. We assume these points are more or less fixed with respect to some of these dimensions (grade level, population density [rural/suburban/urban], demographics, region, etc.) but we like to believe that we can change the position of these points with respect to other dimensions (retention, discipline, standardized test scores, etc.).

Why is it so important to think in terms of this multidimensional space? Because there are few meaningful statements that are valid across these various axes. When Doug Staiger and Jonah Rockoff (here via Ray Fisman) made radical suggestions about teacher hiring policies, they based them on a study of arguably the two least representative school districts in the country. Even if the rest of the study were sound (rather than being a train wreck, but more on that later), the findings would be worthless for most of the country.

Worse yet, when you make a substantial change in educational policy, there is a wide range of relationships between the effects you see along different dimensions, including possible inverse relationships between retention and other measures of school performance (the fastest and most reliable way for a school to improve its performance is to get rid of the students it can't handle).

Even with the most careful of reasoning, the most clearly stated questions and the most closely examined assumptions, this kind of complex, multidimensional system can react to new conditions in dramatic, counterintuitive ways. If you approach it with the kind of sloppy thinking that has dominated the education debate, you are asking fate to do some very bad things.


[thanks to Wikipedia for the hypercube]

A dark, swirling, mammoth wall of wrong

Late last night (or more accurately early this morning) I had the TV on as background noise as I debugged some text mining code. The late show was airing Hidalgo and I happened to tune in shortly before the sandstorm scene.

Today, as I read this post by Ray Fisman, I had the sensation of being engulfed, much as the unlikely riders were, in an enormous, violent impenetrable cloud of bad arguments, flawed reasoning, shoddy research and statistical errors.

I'll try to make some sense of this tomorrow but in the meantime, check out Joseph's comments here.