Bias versus precision

In epidemiology, we are typically trying to estimate an unbiased measure of associations between an exposure and an outcome. Generally, we punt on the causal question of "does exposure X cause outcome Y?", but it is inevitably in the background. After all, if we say that poor exercise habits are associated with early mortality it is generally taken as an advisory to consider improving one's exercise habits rather than as an interesting coincidence.

But not all models are confounding models and the instincts that serve us so well for confounding models can be misleading for predictive models. Nate Silver has a very well explained example of how inaccurate (or, to be more formal, imprecise) predictive models can be worse than biased models. It's a very interesting confusion between bias and precision but it makes me wonder if we don't focus too much on unbiased and too little on efficiency for some of our models.

Thoughts?

More on creativity and slippery metrics

John D Cook has a interesting take on the whole creativity question. I'm not sure that I accept the premise that schools have as much influence on these factors as people think. But it does bring up an interesting point of the tail wagging the dog.

On of the classic criticism of scientific management is that they took a good idea (try and find measureable metrics of business success and failure) and turned it into a myopic focus on what is easy to measure in a spreadsheet. As a result, they focused on easy to measure metrics of success and, in the process, tended to neglect things that are hard to measure.

It's obviously true that creativity is a slippery and hard to measure concept. It is hard to, for example, design a standardized test around it. I wonder if part of the issues of education reform come from a focus on what can be measured and not what is important?

The best way to get rich is to sell books to gullible people

I sometimes wonder how much the current crisis was acerbated by bad advice from financial gurus on TV, advice that has changed amazingly little. Here's a recent example from David Bach (author of Start Late, Finish Rich and endless variations):

The best type of debt is debt that builds wealth over the long run, and the
No. 1 example of that is mortgage debt.

"Home values have increased an average of 6.5 percent a year over the past 30 years," says Bach. "So when you borrow to buy a home, chances are that's good debt. You'll build value."

Bach heavily promotes the idea of homeownership, saying that everyone needs to own where they live. "About 40 percent of Americans are renters," says Bach, "and the fastest way to wealth in America is buying where you live."

Bach cites some shocking numbers to back this up. "The average renter has a median net worth of $4,000, and the average homeowner has a median net worth of about $150,000."

...

One of the reasons so many Americans seem mired in bad debt (Bach reports that the average American carries approximately $8,400 in credit card debt) is that financial education is pratically (sic) nonexistent. "This type of commonsense stuff isn't taught in school," says Bach, "and most Americans don't realize how bad high-rate credit cards are hurting them."

Bach has been doling out this same advice complete with the same net worth statistic (apparently included under the assumption that schools didn't teach statistics either) for about a decade*. It is slightly more sound now than it was five years ago, but it is still not what you'd call good.

This ground has been covered before (notably by Felix Salmon) but just to recap:

1. Home ownership is not a particularly good investment either in terms of returns. It's bad in terms of liquidity and horrible in terms of diversification;

2. Analyses purporting to show the amount saved by owning vs. renting almost always overlook the direct and indirect costs associated with suburban commuting.

3. Home ownership makes it difficult for workers to go where the jobs are. This has economic consequences on the national level, but it's on the individual level that this can turn into a real horror show. Housing prices, liquidity and employment are all correlated by region.




*In Start Late, Finish Rich, Bach says "you cannot get rich being a renter."

Creativity

Felix Salmon has a post on creativity that got me thinking about comparisons. After all, in 1966 (when the test that he discusses was developed), it was a lot harder to get access to first rate creative materials. In that sense, "do it yourself" made a lot of sense and might have encouraged creativity.

You can see this as technology progresses. Consider live music and plays over time. In 1890, if you wanted to hear music or see a story acted out in your home town then you needed to get a local musician to play or go to a local community theater. In 2010 you downloaded the music to your iPod or a top film to your television set.

You see this phenomenon with lots of things. When my dad did weight lifting in the 1960's, the best you could do was a book on the topic. People competed to invent new techniques (and likely had a lot of injuries as a result). Today, I can find massive amounts of information on the topic sitting at home. In the 1970's, Gary Gygax and company could create a new type of game (role-playing games) as an extension of war gaming. Today the ability to create a new game is much harder given the development costs.

So is the lack of creativity a symptom of a richer environment where finding rhea optimal solution is better than creating a new (and likely sub-optimal one)?

I am not saying that this is the case and Felix Salmon brings up a lot of other good points. But it's another reason to wonder about what the implications of these claims would be, even if they were to prove to be correct.

Net impact

The ever interesting Professor in Training has a post on the issues of extraneous demands on a professor's time. I think that this is another example of things where each individual element is likely to be justified but the whole is not.

Examples of these sorts of things include regulations, old laws (regarding horse thievery, for example) assistant professor committee work, software patches and so forth. For example, the last university I was at required a one hour course on the hazards of asbestos every year. While I am sure that asbestos is deadly substance, it seems like overkill to have a class on such a narrow topic every year.

What is hard to determine is how to identify these situations ahead of time and determine the appropriate responses. It's not easy -- as a junior professor I have said no to a few requests already that were pretty reasonable (and sounded like they'd add value) just to make sure that I had enough time.

I would like to think of a way to handle this issue that did not involve "over-all impact committees".

Another reason not to aggregate elementary and post-elementary school data

This is from a National Center for Education Statistics summary of TIMSS data from the Nineties* when the reform movement was pretty much settling into its present form. They were already sounding the alarm about math and science scores and proposing extensive reforms for grades K through 12, despite the fact that the performance issues were almost entirely limited to junior high and high school.

In mathematics, fourth-grade students in 7 countries outperform our fourth graders (Singapore, Korea, Japan, Hong Kong, the Netherlands, the Czech Republic, and Austria). Students in 6 countries are not significantly different from ours (Slovenia, Ireland, Hungary, Australia, Canada, and Israel). U.S. fourth graders outperform their counterparts in 12 nations (Latvia, Scotland, England, Cyprus, Norway, New Zealand, Greece, Thailand, Portugal, Iceland, Islamic Republic of Iran, and Kuwait).

In science, students in only one country--Korea--outperform U.S. fourth graders. Students in 5 countries are not significantly different than ours (Japan, Austria, Australia, the Netherlands, and the Czech Republic), and U.S. fourth graders outperform their counterparts in 19 nations (England, Canada, Singapore, Slovenia, Ireland, Scotland, Hong Kong, Hungary, New Zealand, Norway, Latvia, Israel, Iceland, Greece, Portugal, Cyprus, Thailand, Islamic Republic of Iran, and Kuwait).

*I'll try to dig up some more recent results just to see how things are trending.

If all students were 15 years old...

Joseph has an interesting link to a comparison of test scores of fifteen-year-olds in different countries. I recommend you go by and take a look. It has some interesting stuff, but what caught my eye was what wasn't there. I couldn't find a complementary break-down for any other age group.

After bad metrics, I think the worst problem in educational research may be the inappropriate aggregation of primary and secondary education. If there was ever a case where two populations should be treated separately it's here. The two systems face different problems of different severity that respond to different solutions.

I'll try to follow up with some specifics later, but in the meantime, when you hear someone making a blanket proclamation about our schools, remember that about the only meaningful statement you can make that's true about primary and secondary schools is that the teachers in both are about to get screwed.

Statistical programming languages

This post by John D Cook got me thinking about whether it is possible to do a similar simplification of programming languages for Epidemiologists doing analysis.

These days I see the following languages in heavy use: STATA, R, SPSS, SAS and some S-plus. Furthermore, one is requires to do data management in some combination of SQL/Oracle, SAS, Excel and Access. That doesn't even touch on the people who still use C++ and/or FORTRAN for specialized programming applications.

My question is which ones does it make sense to support? In my department, I think we'll do a combination of R (freeware, flexible, powerful) and SAS (FDA standard) as languages that we officially support. But not supporting STATA is a very painful choice! What have others done in similar circumstances?

Speaking for the unhinged

Jonathan Chait dismisses critics of proposed education reform as 'unhinged.' Speaking as one of the whackjobs, here's the sort of thing that makes us loonies so nervous.

From a press release (6/8/09) from the Department of Education:

Emphasizing the need for additional effective education entrepreneurs to join the work of reforming America's lowest performing public schools, U.S. Secretary of Education Arne Duncan told reporters during a conference call this afternoon that states must be open to charter schools. Too much is at stake for states financially and for students academically to restrict choice and innovation.

"States that do not have public charter laws or put artificial caps on the growth of charter schools will jeopardize their applications under the Race to the Top Fund," Secretary Duncan said. "To be clear, this administration is not looking to open unregulated and unaccountable schools. We want real autonomy for charters combined with a rigorous authorization process and high performance standards."

From a May 1st story in the New York Times:

But for all their support and cultural cachet, the majority of the 5,000 or so charter schools nationwide appear to be no better, and in many cases worse, than local public schools when measured by achievement on standardized tests, according to experts citing years of research. Last year one of the most comprehensive studies, by researchers from Stanford University, found that fewer than one-fifth of charter schools nationally offered a better education than comparable local schools, almost half offered an equivalent education and more than a third, 37 percent, were “significantly worse.”

Us nutjobs would like to see the administration reconsider its position based on the data.

"Fewer Low-Income Students Going to College"

From the Wall Street Journal blog Real Time Economics (via Thoma):

Fewer low- and moderate-income high school graduates are attending college in America, and fewer are graduating.

Enrollment in four-year colleges was 40% in 2004 for low-income students, down from 54% in 1992, and 53% in 2004 for moderate-income students, down from 59% over the same period, according to a report recently submitted to Congress by the Advisory Committee on Student Financial Assistance.

If that trend has continued, low- and moderate-income students who don’t move on to college face an even darker outlook. The unemployment rate for 16- to 19-year olds averaged 17% in 2004, the jobless rate for people over age 25 with just a high school diploma averaged 5% the same year. So far this year, those figures have jumped to 25.8% and 10.6%, respectively.

College expenses and financial aid have become increasingly larger considerations for parents and students, driving more qualified students away from enrolling in four-year colleges.

Harm Reduction

We've been a little bit into economics and education lately, so I suppose here is a good chance to look a publich health issue of interest. In today's British Medical Journal, Kimber et al. report on a prospective cohort study looking at opiate substitution in injectable drug users.

The findings were dramatic:

For each additional year of opiate substitution treatment the hazard of death before long term cessation fell 13% (95% confidence interval 17% to 9%) after adjustment for HIV, sex, calendar period, age at first injection, and history of prison and overdose.

This is a massive effect. Just one year of treatment is competitive with interventions that we would not even think twice about: many drug treatments for cardiovascular disease fall into this range. Since we can't really randomize people to these types of treatments, the prospective cohort design is likely as good as it gets. So perhaps these types of dramatic findings, if properly replicated, can start a discussion about harm reduction once the behavior is initiated. After all, there is a large population of injectable drug users and finding ways to improve their (fairly poor) outcomes seems like a decent way to improve public health.

More on tenure

In a thread on DrugMonkey, a commenter posed this solution to tenure:

The society as a whole would benefit from abolishing tenure system and putting educators where everyone else belongs, a market place.

As you may guess, I had some thoughts about this comment. Markets are pretty amazing things but they do some things rather poorly. Mark Thoma has a good discussion of the criteria for a market to work efficiently:

http://economistsview.typepad.com/economistsview/2007/02/markets_are_not.html

Research is, in some important respects, a public good and, like other public goods is hard to place into a market easily (in the same sense that is hard to privatize the road system). Furthermore, the NIH style grant system is an attempt (so far as I can tell) to handle the information problems with basic research in the most rational way possible under the circumstances.

This is not to say that change and reform are not possible; they are. But I find it odd that tenure is attacked whereas other forms of job security are not (for example, there is no widespread call to privatize the military). The university and NIH system are designed to introduce as mush competition into research and teaching as you can with a single buyer dominating the marketplace (that would be the federal government, via student loans and research grants).

I think tackling this requires a series of marginal improvements. Standing on the outside it is easy to suggest radical change but what do you do if the radical change results in a worse outcome than the current state of affairs? In particular, I want to see more about how to apply a market based system to higher education given all of the barriers to an efficient market that are present.

Post tenure era?

DrugMonkey has a post today called "A Farewell to Tenure". It's an interesting post in the context of Mark's discussion about the ending of teacher tenure in the K-12 system.

However, I think that the major driver of the reason that tenure can be attacked in the academic world is that there is a dual career track for many academic disciplines. The fields with lots of soft money jobs (employment based 100% on external grant funding) tend to be focused on research topics that are seen as priorities. Here there are specific metrics (such as key scientific papers published and novel ideas proposed) that make it possible to actually hold competitions for funding.

What is the analogous parallel funding track for K-12 teachers?

Thinking about teacher layoffs -- the logistics of staffing

Regarding Seyward Darby's recent article on teacher layoffs, you can find posts on why she doesn't understand economics here and here, posts on the questionable data and analysis behind her position here and here, posts pointing out the problems with the movement she endorses here and here (from Darby's own magazine) and an explanation of why the tenure 'reform' she champions would be unfair and unethical here.

But I don't believe anyone has pointed out what a logistic nightmare she wants to get us into.

Let's take the plan that Darby favors and throw in some reasonable numbers:

We take our already somewhat understaffed schools and lay off, let's say, 200,000* teachers selected based in part on some quality metric. We then run the schools severely understaffed for the next year or so until state revenues recover.

That 200K will be made up of three groups, the good, the bad and the better-than-nothing. The good are effective teachers who end up on the list through a combination of bad luck, bad metrics and bad administrators (go here to see how the last two can work together). The bad are teachers at the very bottom of the quality scale. The better-than-nothing are teachers who aren't all that effective but are probably still at least as good as most of the people you could get to replace them.

Given the flaws in our system of ranking teachers and the innate difficulty of the problem, a fair number of good teachers will end up on the list. Likewise, given the problems with recruiting teachers for problem schools, better-than-nothing teachers will continue to make up a large part of it.

The upshot of all this is that, given population growth and the high level of teacher attrition, you will need a substantial number of the people you laid off to come back if you are to have any hope of meeting staffing targets. This wasn't as much of a problem under last-in/first-out for a couple of reasons. First, the attrition rate for new teachers was so high that many of the teachers you laid off wouldn't have been there in a couple of years even if you hadn't let them go. Second, leaving under last-in/first-out carries minimal stigma. For those who really wanted to it was easy to get back into the field a year later.

Under the 'reform' system, there is a serious stigma and a deadly asymmetry of information problem. Keep in mind that administrators are basically stuck with new hires for a year (you try finding a certified replacement in, say, November). They know that a teacher laid off under that system might turn out to be first rate, but do they really want to take the chance?

Let's say 100,000 of the laid-off teachers fell into the need-them-later category. We have screwed these teachers out of contractually obligated compensation, scapegoated them for all the problems in the education system and made them unemployable in their chosen field. Would you come back under those circumstances?

Of course, the impact is not limited those laid off. Other than the satisfaction that comes with the job, the primary attraction of teaching is job security. For generations, teachers have self-selected to be risk-averse. Faculty rolls mainly consist of people who have willingly traded any significant chance of promotion for the stability to start a family, buy a house, and spend thirty or forty years doing something they're good at.

In an industry dependent on a very large number of highly skilled labourers, this is not a bad situation, but putting those bigger, long-term questions aside, what is likely to happen when the economy recovers and we try to start restaffing? We were having trouble maintaining adequate levels before the lay-offs. Then we permanently removed around 200,000 teachers from the pool (many of whom we would have liked to have had back later). Now we are trying to hold on to teachers after having unilaterally removed a valued part of their compensation and established the precedent that contractual obligations to educators don't need to be honored.

Of course, the reformers will assure you that they have a set of programs that will solve the retention problem, but keep in mind these are the same people who have been solving this problem for a decade and their track record is nothing if not consistent.

According to this NCES paper, we were hiring around 200,000 new teachers a year before the economy collapsed. That was a difficult and expensive proposition and we often had to compromise quality to make that number. Now think about doubling or tripling the number of vacancies.

This is not going to work out well.







*without the plan that Darby thinks might merit a veto, 100,000 to 300,000 teachers could lose their jobs.

Wayward Seyward

I mentioned in my previous post that when I saw Seyward Darby had written an article entitled "Obama: Don't Touch Education Reform," I had a bad feeling. Bad and, as it turns out, damned accurate:

But this particular measure has evolved in some decidedly unappealing ways. When it was first introduced, numerous education advocacy groups asked Congress to impose constraints that would effectively compel states to reform their layoff systems. As I explained recently, most states and school districts follow a misguided "last-hired, first-fired" rule. If they must let teachers go for budgetary reasons, they start by booting those teachers who have spent the least time in the classroom. Teacher quality doesn't factor into the decision at all. Congress, though, refused to go along with the proposal—and the Obama administration didn't intervene. "Right now, the most important thing is to stop the bleeding," Senator Tom Harkin, chair of the Health, Education, Labor, and Pensions Committee, said in early May.

That was bad enough. But then, this week, House Appropriations Chairman David Obey proposed that the government pay for the legislation, in part, by taking money from other Education Department funds—and not just any old funds, but the money set aside for some of Obama’s most important school reforms. Under Obey's plan, which the full House ultimately adopted, $500 million would come from Race to the Top, a competitive grant program and probably the most talked-about aspect of Obama's education agenda; $200 million would come from the Teacher Incentive Fund, which supports performance-based compensation plans; and another $100 million would come from money for charter schools. Sounding a bit like Harkin, Obey reportedly said, "When a ship is sinking, you don't worry about redesigning a room, you worry about keeping it afloat."

...

And if the push for other funding fails? The administration will have a tough choice to make. But one thing's for sure: Letting Congress chip away at the education reform agenda now would place it on a slippery slope in the future. And that's something the country can ill afford.

"Tough choice"? Let's review:

1. We are on the verge of adding a Roman numeral to the end of the Great Depression. Under these circumstances, mass layoffs of teachers is insanely pro-cyclic. It doesn't matter which teachers you pick, the immediate economic effect is the same. Even if you had a reliable metric that could identify poor teachers, this would be the worst possible time to get rid of them;

2. You don't have a reliable metric that can identify poor teachers;

3. Much of the threatened reform package is actually a continuation of policies that have a long (sometimes decades-long) and horrible track record.