One final(?) reason not to trust teaching metrics

Don't be too afraid (or perhaps relieved) at the thought of OE dropping the subject of education and education reform. I think I can speak for Joseph here when I say that neither of us plan to drop the topic until pundits stop saying stupid things about it.

What follows might be a final reason because I think we've got a pretty good framework for discussing the dangers of making radical changes based on the proposed metrics for measuring teacher performance. Do a keyword search on education to see what I mean.

A few months ago, Joseph and I had a marathon phone conversation where we sketched out the attributes a well designed study of teacher performance would have to have in order to cope with self-selection, interactions, social dynamics, nesting and the myriad other challenges that go with this problem. You will occasionally find an educational study that attempts to deal with one or two of these challenges but, as far as I can tell, no educational study address all of them.

Some of these problems have been widely discussed. Others have been examined in some detail in this blog. But there is at least one that we haven't gotten around to: the question of how certain teachers mesh with certain classes.

Every teacher, no matter how skilled or experienced, will do better with some types of classes than with others. Different classes have different needs. Some need a fast pace; some need patience; some need freedom to explore; some need structure and discipline. After you've been teaching for a while you learn to read classes and adjust your style and presentation but no teacher is equally good under all conditions.

This raises two serious problems for the educational researcher: first, how do you assign classes to teachers and students to classes in a way that protects you from aliasing problems?; second, how do you interpret the results?

Here's a example, let's say teacher A does well with remedial math but does badly with calculus. Teacher B is great with calculus but simply can't communicate with the remedial kids. How do you score those teachers? You could use the mean and conclude that both teachers were doing an average job -- a conclusion that is pretty much wrong in all four cases. You could take the higher of the scores, working under the assumption that administrators are competent managers and therefore know enough to put teachers in classes that match their abilities. Or you might decide that since the purpose is to catch bad teachers we should take the low score. Or you could pick calculus because it's more advanced. Or you could pick remedial because it affects those kids most likely to be left behind.

Of course, all of this presupposes that we have this kind of information available when we try to measure teacher performance (which we never do). It also assumes that we want to have a serious discussion using meaningful data, that we want to be honest and fair, and that we actual care about the quality of our kids' education.

Perhaps it's just the lateness of the hour and the weight of a long week, but I find it increasingly difficult to hold onto that last set of assumptions.




(I'm writing this under the gun and corners are being cut. I will try to go back and add links when the storm passes.)

Variable definitions are tricky

Tyler Cowen calls this the blog post of the year and I am inclined to agree. It gets at a classic problem of labeling of variables that measure a lot of different things. In this case, the scale that is being evaluated measures both the size of government as well as the quality of government. It's hardly a shocking discovery that low levels of corruption are good for economic growth. But it leads to very different policy conclusions than a discovery that large government is simply bad.

[to be fair these factors may be even more correlated than the analysis suggests as it might be easier to police corruption in a small government]

Once again, this is a lesson that we should take to heart when considering complex diseases in epidemiology. Low adherence to medication could be driven by factors like access to care or being health conscious. But it could also be a proxy for which populations show a higher burden of uncomfortable side effects. Disentangling these complex activities (like decisions to remain adherent) isn't trivial and can lead us down very misleading pathways if we aren't critical.

Professor Cowen's pointer is well taken.

Association, contribution and causation

This piece (on the contributions of medical expenses to bankruptcy) has so e important lessons for epidemiology as well. It's easy to forget that a "risk factor" may be neither a sufficient nor a necessary cause of a clinical endpoint.

For example, smoking is associated with increases in the rates of myocardial infarction. However, some people may smoke heavily for a lifetime and never experience the endpoint (despite living to an old age). Conversely, a smoker who suffers a myocardial infarction may have been fated to have the event anyway. It's even tricky to really estimate the attributable fraction of the risk as confidence intervals are often broad and we are never certain that the conditions for an unbiased estimate (namely, no unmeasured confounders and a clear counter-factual case) are present.

So saying that exposure X contributes to the outcome Y does not make the strong claims of causality that it might seem on first glance. In fact, exposure X can be an real predictor and be increasing but not explain much of the variability in Y. Even worse, there can be issues of meqasurement error. In the financial example, what if a medical expense were to be placed on a credit card? In the smoking example it is clear that estimating pack-years of exposure is likely to be very approximate with typical levels of record-keeping.

So it's not trivial to assess the absolute importance of associations -- even if they are shown to exist in data.

How education reform is becoming like the Eurozone

One of the great lessons that can be (though probably won't be) learned from the current economic crisis is that you shouldn't force yourself into a situation where you have to apply a single solution to a diverse set of problems. Much of Europe's current woes can be traced back to the fact that countries like Greece have to have the same monetary policy as countries like Germany. (Do a search for the Euro on Paul Krugman's blog for a detailed explanation of why this is such a bad idea.)

Schools are also a diverse population where elementary schools have a completely different set of challenges than do the higher grades, and where urban, suburban and rural schools each face radically different problems. Even if the proposed reforms are a good fit for one group of school (and that's far from a safe assumption), they will certainly be a poor, perhaps even disastrous fit for others.

Tenure: I just can't look away . . .

Another couple of good Tenure posts. One from Megan McArdle's blog by rudidiri:

Two points: (1) the worldwide most successful Universities have the tenure system; (2) higher education is no less competitive than other businesses. Harvard and Chicago compete with Universities worldwide for students and researchers. If the tenure system is so terrible how come nobody has entered this market and made it to the top?

If someone complains about executive compensation then libertarians are quick to point to market forces. How is this different? When it comes to Universities, libertarians all of a sudden fret about the contract choices of organizations in a highly competitive market. Could it have something to do with the political views of the University employees? Of course not. That would reveal an embarrassing lack of self-awareness.

This point has flaws as there may be strucutral issues that prevent corporations from moving into the Higher Education market as the conditions for an efficient market are almost certainly not met. However, it's also unclear that a sufficiently large pension plan for a senior executive is all that different in kind than tenure for an unproductive professor.

The idea of tenure as compensation is even more explicitly made by mathgrl:

I am ambivalent about tenure, but its removal does have some implications:

1) With tenure, I am free to pursue high risk research. Each project usually takes 1 to 3 years. With high risk, potentially important research, there is a good chance that I spend that time and have little to show for it... or I could change my field (which may affect your energy costs, carbon footprint, etc). The trend in research labs (notably Bell)--which do not have tenure--has been toward quarterly progress reports. They kill good, long term projects. With the removal of tenure, I could see universities heading down this path.

2) You will have to pay me much, much more. I work for far below market rates because I have the promise of security (tenure) and the ability to work on what interests me (to the extent that I can find grants to support this work). Right now, I work the same number of hours as an i-banker at 33-50% of the wages, while having superior training. If I am working every day in fear of being fired, you will have to pay me about 70-80% of corporate rates (on par with research labs).

Are those negatives worth it? I am not sure. The removal of tenure may be fine for some teaching posts (cf adjuncts) but it might cause long term problems if it is removed for research posts (higher education costs, lack of basic R&D).

I also made the decision to take a tenure track position. It actually was not a trivial decision to make as there were a lot of tough considerations. But it's absolutely clear to me that I'd have taken a corporate job in the absence of either tenure or a very long term employment contract. The ability to work on long term projects is the main benefit to me of academia and that's not going to be possible if only short term contracts exist.

It is also the case that the academic market is far from homegenous and it's fairly dangerous to make broad, sweeping statements about the employment practices across such a broad range of fields and institutions. But, that being said, it's odd that it is in vogue to attack employment contracts all of a sudden.

New classes of analgesics

There is an amazing post by JuniorProf on this subject, Go and read.

The truth is that while there are effective classes, all of them have one of three potential issues:

1) Cognitive fogging and addiction

2) Increased risk of gastrointenstional bleeding and/or myocardial infarction and/or fractures

3) Liver toxicity

Now, because pain is hard to measure and these other outcomes are easy to meaure, it is easy to end up under-treating pain in order to reduce the rate of hard outcomes. Needless to say, this is a dreadfully sub-optimal treatment equilibrium. Even worse, these cases can lead to criminal charges when errors occur or even just due to cocnerns over the large doses that some patients require for routine management.

Clearly a new class of pain medications would be a solution to these issues. There is also, however, a cultural piece, that involves the stigmatization of people in legitimate pain that could be addressed.

All in all, an area that really deserves more attention.

Teaching yourself mathematics -- a footnote for future posts

I'm pretty sure I'm going to be making this claim repeatedly so I might as well take a few minutes to put it down in a linkable form for future use.

Of all the subjects a student is likely to encounter after elementary school, mathematics is by far the easiest to teach yourself. With the appropriate attitude and assumptions, adequate motivation and a simple and easily mastered set of skills the majority of students can take themselves from pre-algebra through calculus.

What is it that makes math teachers so expendable? Part of the answer lies in mathematics position on the fact/process spectrum. Viewed in sufficiently general terms, all subjects start with giving the student a set of facts and ideally end with the student performing some process using those facts. In subjects like history and to a slightly lesser extent, science, most of the early stages of mastering the subject center around absorbing the facts. On the other end of the spectrum, subjects like music, writing and mathematics involve a relatively small set of facts*. Students studying these subjects tend to focus primarily on process almost from the beginning.

Put another way, at some point all disciplines require the transition from passive to active and that transition can be challenging. In courses like high school history and science, the emphasis on passively acquiring knowledge (yes, I realize that students write essays in history classes and apply formulas in science classes but that represents a relatively small portion of their time and, more importantly, the work those students do is fundamentally different from the day-to-day work done by historians and scientists). By comparison, junior high students playing in an orchestra, writing short stories or solving math problems are almost entirely focused on processes and those processes are essentially the same as those engaged in by professional musicians, writers and mathematicians.

Unlike music and writing, however, mathematics starts out as a convergent process. It doesn't stay that way. By the time a student gets to upper level math courses like real analysis or applied subjects like statistics ** there are any number of valid proofs for theorems and approaches to problems, but for anything before, say, differential equations, most math problems have only one solution and students are able to quickly and accurately check their work. Compare this to writing. There is no quick or accurate way to gauge the quality of a piece of prose or, worse yet, verse. Writers spend years refining their editing skills and even then they still generally seek out other critics to help them assess their own work.

This unique position of mathematics allows for any number of easy and effective self-study techniques. One of the simplest is the approach that got me through a linear algebra section taught by the worst college level instructor I have ever encountered (and that, my friends, covers some territory).

All you need is a textbook and a few sheets of scratch paper. You cover everything below the paragraph you're reading with the sheet of paper. When you get to an example, leave the solution covered and try the problem. After you've finished check your work. If you got it right you continue working your way through the section. If you got it wrong, you have a few choices. If you feel you basically understood the solution and see where you made your mistake, you might simply want to go on; if you're not quite sure about some of the steps in the solution, you should probably go back to the beginning of the section; if you're really lost, you should go back to the preceding section and/or the previous sections that introduced the concepts you're having trouble with.

Once you've worked through all the examples, start on the odd numbered problems and check your answers as you go. If you're feeling confident, you can skip to the difficult problems but if you make a mistake or get stuck you should probably go back to number 1.

Don't get me wrong. I'm not saying this is the only technique, let alone the best, for teaching yourself mathematics. Nor am I suggesting that we make a practice of dumping student in sink-or-swim situations. I think we should provide students with the best teachers and support system possible, but even under those conditions, the internal resources needed to teach yourself mathematics are tremendously valuable to all students and are absolutely essential to anyone who has to use sophisticated analytical reasoning.

Tragic postscript: In what I can only assume is an idiotic attempt raise standards, most books have stopped giving answers to odd-numbered problems. Under the old system you would assign odd problems when you wanted the students to be able to check their work and even problems when you wanted to make sure they weren't just looking answers up in the back of the book. It was a simple, flexible, and effective system that encouraged students to be independent and resourceful. No wonder it was such a prime target for reform.

* I heard a story (possibly apocryphal) of a professor who walked into an upper level math class, wrote properties of real numbers on the board, told the class that was all they needed to know to prove all the theorems in the book, then walked out.

** The relationship between mathematics and statistics is particularly complex, far too complex to discuss in a blog post.

Marginal Revolution has amazing comments

This has led me to quote a few of the best points made in the latest discussion on tenure. From PQuincy:

By the way, I do admit that tenure (which I, like Tyler, possess), may lead to the payment of rents, and to some rent-seeking behavior. But if anyone claims that, say, the modern financial sector involves no rents and rent-seeking, I'll laugh out loud. The better question is to ask what the negative burden of rents in universities is compared to potential benefits of a system that awards some kind of tenure. Given that few university professors make very much money, my first guess (no doubt shaped by self-interest) is that tenure is a relatively inexpensive way to achieve certain hard-to-value-but-real outcomes.

This does make a point that is often absent from the debate-- when we say that tenure is a form of rent-seeking is it really the current highest priority? After all, financial services are able to create all sorts of captive income streams (think of fees on 401(k) accounts) that are both more expensive and harder to justify.

From Cliff Bekar

Evaluating a good hire into an academic department is expensive. It would be very expensive for managers (Deans, Provosts, etc.) to evaluate the contributions of an academic to a literature. Hiring an outside firm would also prove a challenge given the huge variance in academic performance and the length of time it takes to determine whether a good hire has been made (it can often take 6+ years for agents to fully realize their research potential). Academics within a discipline have a comparative advantage in making such determinations (as an economic historian I would be at a significant disadvantage in evaluating, say, a new econometrician). Further, since the quality of their department stands to gain the most from making a good hire, academics within the department have an incentive to maximize the quality of hire. But imagine that the people in charge of the hire may be fired. In practice decisions to fire an academic will always be made in part relative to the standards of a department/institution. An agent that is performing poorly relative to their colleagues is more likely to be let go. A 50-year old academic, who might lose their job if faced with high performing colleagues faces perverse incentives in the hiring process. Tenure renders the agents who are best able to evaluate talent with the good incentives to hire the best possible fit. This effect is even more important if we consider other elements of an academic hire that make "insiders" better at evaluating potential fits: What types of courses would the existing student mix best benefit from, what comparative weaknesses in the curriculum should be addressed with a new hire, what is the culture of the department? Good answers to these questions are often hard to answer by simply accumulating the kind of data a third party (recruiting firm or Dean) could cheaply secure.

This is a fantastic point and really shows how tenure could work to increase excellence. Now, it is true that this argument worked a lot better with a mandatory retirement age (as end of career issues are proving sticky) but that seems like a possible point of intervention, doesn't it? Making tenure no longer automatically protect jobs after the eligibility age for social security would be the example of a small tweak that radically improves the system.

Finally:

No institution is perfect; tenure is no exception. But it is a way to allign faculty interests with the long term interests of their institutions. In this era of 5-year-turnover-administrators, without tenure what makes anyone think that short term administrators will do what is in the long term interest of their current institutions?

It's hard to align interests effectively given short term windows (industry constantly grapples with this issue) and tenure seems like a cheap way to do it in a very strong way.

It's not an accident that Marginal Revolutions was the blog that first got me to read blogs. Very insightful stuff.

Seyward Darby -- now not even pretending to think things through

The idea that the proposed educational reforms will magically fix everything has become so ingrained in Darby's thinking she has started writing things like this:

Will states, school districts, and unions also agree to revise, in the long term, the ways teachers are hired, compensated, and fired, hopefully precluding the need for mass layoffs in the future?

Even if there weren't serious problems with the proposed metrics for compensating and firing teachers, even if the changes were to do everything the advocates have claimed, how would these changes have any effect on the pro-cyclic economic policies of states? Has anyone even suggested a relationship here?

And if we're not limited to cases where there is some sort of likely causal relationship, why limit ourselves to lay-offs? Why not have something like this: "Will states, school districts, and unions also agree to revise, in the long term, the ways teachers are hired, compensated, and fired, hopefully ending future violence in the Middle East, curing cancer and preventing future movie version of SNL sketches?"?

What the hell, dream big.

A quick note on tenure and the economics of deferred compensation

(Joseph may be through with this, but I'm just getting started.)

One of the many weird things about the exceedingly weird tenure debate is the way many (hell, most) of the participants talk about tenure as a gift to teachers rather than a form of compensation that's part of a highly competitive market. *

All forms of deferred compensation offer two attractive features to employers: they reduce the initial cost and they encourage retention. Both of these features are particularly attractive to school administrators who work in an incredibly labor-intensive industry, can't offer big salaries, often require employees to move to out-of-the-way towns and can't afford to have their best people move on the moment a better offer comes along.

If you eliminate tenure you will have to come up with an alternative compensation plan that accomplishes the same thing and with the possible exception of Tyler Cowen, no one seems to show any interest in what that alternative might be.




* Of course you can also talk about tenure as a way of protecting academic freedom and discouraging grade inflation, though no one does anymore.

One last post on Tenure

I am not surprised that Tyler Cowen is asking the hard question that is at the center of the tenure debate. Namely, what is the plan forward if we replace tenure.

In particular, he asks for a concrete number for how the wage equilibrium shifts if tenure is removed:

If firing is in order, how much higher do initial wage offers have to be?

Megan McArdle gave a "toy example" that seemed more rhetorical than actual. In point of fact, moving to a small university town as a highly educated professional might be a major decision. True, in some disciplines there is a real case for a criminal over-production of PhD-level credentials. But it seems odd that the solution to this problem is to make the one part of the job chain that is actually good into something that is worse.

I'm always delighted by Tyler Cowen's insights and recommend reading the whole thing.

Paul Krugman and the limitations of historical analysis

I really like this post from Paul Krugman both for the reasoning and, more importantly, his awareness of the limitations.

So, about the first point: economic history is a great source of evidence about how the economy works – in fact, pretty much our only source. And the RR project, drawing on evidence from much further back and farther afield than usual among economists, was a great idea.

But there are usually major problems with historical analysis, no matter how much data you have, because it’s very difficult to isolate the things you’re interested in. There’s an old line to the effect that everything in the economy affects everything else, in at least two ways; this gives enormous room for spurious correlation. Econometrics is supposed to provide ways to disentangle the effects of multiple factors, but it’s difficult, and my sense is that few big arguments in economics have ever been settled by multiple regression analyses, let alone by all the sophisticated techniques developed these past three generations.

But Reinhart-Rogoff is relatively robust to these problems. Why? Because it focuses on extreme events. Financial crises are very big things, sharply concentrated in time. As a result, it’s reasonably certain that the economic developments in the aftermath of a financial crisis were driven by that crisis, not by other stuff that may have been going on.

Of course, not all economists are so careful when it comes to drawing their conclusions.

More on Tenure

I usually really like DeadDad. But this line in his post today:

It’s about recognizing that the tenure system feeds the adjunct system, and that the only way to get rid of the latter is to get rid of the former.

seemed to strike a really odd note. It seems to be assuming that if you removed job security (aka tenure) from the people with benefits then you would see adjuncts being treated better. This statement assumes a couple of things. One, is that there is a pile of resources locked up by older professors who refuse to work or retire. Two, it assumes that (in no sense) can we consider a gradual fading out of the job to be a form of delayed compensation for the early years of fighting for tenure. I am not sure what it is about deferred compensation that makes people want to break agreements about it so badly but it's certainly a new epidemic.

Finally, he assumes that a surplus generated by reducing the costs of tenure would not be eaten up by: a) administration, b) higher compensation to star professors to account for lower job security or c) reduced funding to the university.

One of the things that tenure works against is the principal agent problem of state agencies: it is hard for a government to regulate bad management without the appearance of corruption. Minimizing the changes that can be done under a single term as a senior administrator (also seen in the civil service) would seem to be a partial solution.

"The Double-blind Job"

I'm not much of a Leverage fan -- if you're in the mood for a caper show I'd stick with Burn Notice and Mission Impossible (the series, not the loud and largely brainless movies) and if you want to see more Gina Bellman there's Blackeyes, Coupling and Jekyll -- but between the title and the setting of the episode below, I just had to post this.

The ASA was out of its mind to give Felix Salmon a reporting award

Is the American Statistical Association even vaguely familiar with the internet? Don't they understand how the blogosphere food chain works? One blogger says something mind-numbingly stupid which causes a wave of outraged and mocking posts in response which causes yet another wave of defences and accusations of hypocrisy.

Obviously the award should have gone to someone like Ben Stein (the phytoplankton of the blogosphere). Stein's writing is filled with the kind of nutrient-rich raw material that bloggers (including Salmon) can write about endlessly.

By comparison, bloggers trying to subsist on Salmon posts are usually left writing things like "Felix Salmon has an interesting analysis of..." or "Salmon makes an important point about..." or "Check out..."

On a related note:

Check out these very amusing posts by Felix Salmon: the first looks at the MBA-Tourette's that afflicts so many online editors; the second has fun with the aforementioned Ben Stein's claim the unemployed deserve to be out of work.

Salmon also has a devastating analysis of CNBC's muddled take on counterfeiting.

And while you're there, take a look at this post that makes an important point about the nomination of Elizabeth Warren for head of the CFPB.

Hell, just read the whole damned blog.