One final(?) reason not to trust teaching metrics

Don't be too afraid (or perhaps relieved) at the thought of OE dropping the subject of education and education reform. I think I can speak for Joseph here when I say that neither of us plan to drop the topic until pundits stop saying stupid things about it.

What follows might be a final reason because I think we've got a pretty good framework for discussing the dangers of making radical changes based on the proposed metrics for measuring teacher performance. Do a keyword search on education to see what I mean.

A few months ago, Joseph and I had a marathon phone conversation where we sketched out the attributes a well designed study of teacher performance would have to have in order to cope with self-selection, interactions, social dynamics, nesting and the myriad other challenges that go with this problem. You will occasionally find an educational study that attempts to deal with one or two of these challenges but, as far as I can tell, no educational study address all of them.

Some of these problems have been widely discussed. Others have been examined in some detail in this blog. But there is at least one that we haven't gotten around to: the question of how certain teachers mesh with certain classes.

Every teacher, no matter how skilled or experienced, will do better with some types of classes than with others. Different classes have different needs. Some need a fast pace; some need patience; some need freedom to explore; some need structure and discipline. After you've been teaching for a while you learn to read classes and adjust your style and presentation but no teacher is equally good under all conditions.

This raises two serious problems for the educational researcher: first, how do you assign classes to teachers and students to classes in a way that protects you from aliasing problems?; second, how do you interpret the results?

Here's a example, let's say teacher A does well with remedial math but does badly with calculus. Teacher B is great with calculus but simply can't communicate with the remedial kids. How do you score those teachers? You could use the mean and conclude that both teachers were doing an average job -- a conclusion that is pretty much wrong in all four cases. You could take the higher of the scores, working under the assumption that administrators are competent managers and therefore know enough to put teachers in classes that match their abilities. Or you might decide that since the purpose is to catch bad teachers we should take the low score. Or you could pick calculus because it's more advanced. Or you could pick remedial because it affects those kids most likely to be left behind.

Of course, all of this presupposes that we have this kind of information available when we try to measure teacher performance (which we never do). It also assumes that we want to have a serious discussion using meaningful data, that we want to be honest and fair, and that we actual care about the quality of our kids' education.

Perhaps it's just the lateness of the hour and the weight of a long week, but I find it increasingly difficult to hold onto that last set of assumptions.




(I'm writing this under the gun and corners are being cut. I will try to go back and add links when the storm passes.)

Variable definitions are tricky

Tyler Cowen calls this the blog post of the year and I am inclined to agree. It gets at a classic problem of labeling of variables that measure a lot of different things. In this case, the scale that is being evaluated measures both the size of government as well as the quality of government. It's hardly a shocking discovery that low levels of corruption are good for economic growth. But it leads to very different policy conclusions than a discovery that large government is simply bad.

[to be fair these factors may be even more correlated than the analysis suggests as it might be easier to police corruption in a small government]

Once again, this is a lesson that we should take to heart when considering complex diseases in epidemiology. Low adherence to medication could be driven by factors like access to care or being health conscious. But it could also be a proxy for which populations show a higher burden of uncomfortable side effects. Disentangling these complex activities (like decisions to remain adherent) isn't trivial and can lead us down very misleading pathways if we aren't critical.

Professor Cowen's pointer is well taken.

Association, contribution and causation

This piece (on the contributions of medical expenses to bankruptcy) has so e important lessons for epidemiology as well. It's easy to forget that a "risk factor" may be neither a sufficient nor a necessary cause of a clinical endpoint.

For example, smoking is associated with increases in the rates of myocardial infarction. However, some people may smoke heavily for a lifetime and never experience the endpoint (despite living to an old age). Conversely, a smoker who suffers a myocardial infarction may have been fated to have the event anyway. It's even tricky to really estimate the attributable fraction of the risk as confidence intervals are often broad and we are never certain that the conditions for an unbiased estimate (namely, no unmeasured confounders and a clear counter-factual case) are present.

So saying that exposure X contributes to the outcome Y does not make the strong claims of causality that it might seem on first glance. In fact, exposure X can be an real predictor and be increasing but not explain much of the variability in Y. Even worse, there can be issues of meqasurement error. In the financial example, what if a medical expense were to be placed on a credit card? In the smoking example it is clear that estimating pack-years of exposure is likely to be very approximate with typical levels of record-keeping.

So it's not trivial to assess the absolute importance of associations -- even if they are shown to exist in data.

How education reform is becoming like the Eurozone

One of the great lessons that can be (though probably won't be) learned from the current economic crisis is that you shouldn't force yourself into a situation where you have to apply a single solution to a diverse set of problems. Much of Europe's current woes can be traced back to the fact that countries like Greece have to have the same monetary policy as countries like Germany. (Do a search for the Euro on Paul Krugman's blog for a detailed explanation of why this is such a bad idea.)

Schools are also a diverse population where elementary schools have a completely different set of challenges than do the higher grades, and where urban, suburban and rural schools each face radically different problems. Even if the proposed reforms are a good fit for one group of school (and that's far from a safe assumption), they will certainly be a poor, perhaps even disastrous fit for others.

Tenure: I just can't look away . . .

Another couple of good Tenure posts. One from Megan McArdle's blog by rudidiri:

Two points: (1) the worldwide most successful Universities have the tenure system; (2) higher education is no less competitive than other businesses. Harvard and Chicago compete with Universities worldwide for students and researchers. If the tenure system is so terrible how come nobody has entered this market and made it to the top?

If someone complains about executive compensation then libertarians are quick to point to market forces. How is this different? When it comes to Universities, libertarians all of a sudden fret about the contract choices of organizations in a highly competitive market. Could it have something to do with the political views of the University employees? Of course not. That would reveal an embarrassing lack of self-awareness.

This point has flaws as there may be strucutral issues that prevent corporations from moving into the Higher Education market as the conditions for an efficient market are almost certainly not met. However, it's also unclear that a sufficiently large pension plan for a senior executive is all that different in kind than tenure for an unproductive professor.

The idea of tenure as compensation is even more explicitly made by mathgrl:

I am ambivalent about tenure, but its removal does have some implications:

1) With tenure, I am free to pursue high risk research. Each project usually takes 1 to 3 years. With high risk, potentially important research, there is a good chance that I spend that time and have little to show for it... or I could change my field (which may affect your energy costs, carbon footprint, etc). The trend in research labs (notably Bell)--which do not have tenure--has been toward quarterly progress reports. They kill good, long term projects. With the removal of tenure, I could see universities heading down this path.

2) You will have to pay me much, much more. I work for far below market rates because I have the promise of security (tenure) and the ability to work on what interests me (to the extent that I can find grants to support this work). Right now, I work the same number of hours as an i-banker at 33-50% of the wages, while having superior training. If I am working every day in fear of being fired, you will have to pay me about 70-80% of corporate rates (on par with research labs).

Are those negatives worth it? I am not sure. The removal of tenure may be fine for some teaching posts (cf adjuncts) but it might cause long term problems if it is removed for research posts (higher education costs, lack of basic R&D).

I also made the decision to take a tenure track position. It actually was not a trivial decision to make as there were a lot of tough considerations. But it's absolutely clear to me that I'd have taken a corporate job in the absence of either tenure or a very long term employment contract. The ability to work on long term projects is the main benefit to me of academia and that's not going to be possible if only short term contracts exist.

It is also the case that the academic market is far from homegenous and it's fairly dangerous to make broad, sweeping statements about the employment practices across such a broad range of fields and institutions. But, that being said, it's odd that it is in vogue to attack employment contracts all of a sudden.

New classes of analgesics

There is an amazing post by JuniorProf on this subject, Go and read.

The truth is that while there are effective classes, all of them have one of three potential issues:

1) Cognitive fogging and addiction

2) Increased risk of gastrointenstional bleeding and/or myocardial infarction and/or fractures

3) Liver toxicity

Now, because pain is hard to measure and these other outcomes are easy to meaure, it is easy to end up under-treating pain in order to reduce the rate of hard outcomes. Needless to say, this is a dreadfully sub-optimal treatment equilibrium. Even worse, these cases can lead to criminal charges when errors occur or even just due to cocnerns over the large doses that some patients require for routine management.

Clearly a new class of pain medications would be a solution to these issues. There is also, however, a cultural piece, that involves the stigmatization of people in legitimate pain that could be addressed.

All in all, an area that really deserves more attention.

Teaching yourself mathematics -- a footnote for future posts

I'm pretty sure I'm going to be making this claim repeatedly so I might as well take a few minutes to put it down in a linkable form for future use.

Of all the subjects a student is likely to encounter after elementary school, mathematics is by far the easiest to teach yourself. With the appropriate attitude and assumptions, adequate motivation and a simple and easily mastered set of skills the majority of students can take themselves from pre-algebra through calculus.

What is it that makes math teachers so expendable? Part of the answer lies in mathematics position on the fact/process spectrum. Viewed in sufficiently general terms, all subjects start with giving the student a set of facts and ideally end with the student performing some process using those facts. In subjects like history and to a slightly lesser extent, science, most of the early stages of mastering the subject center around absorbing the facts. On the other end of the spectrum, subjects like music, writing and mathematics involve a relatively small set of facts*. Students studying these subjects tend to focus primarily on process almost from the beginning.

Put another way, at some point all disciplines require the transition from passive to active and that transition can be challenging. In courses like high school history and science, the emphasis on passively acquiring knowledge (yes, I realize that students write essays in history classes and apply formulas in science classes but that represents a relatively small portion of their time and, more importantly, the work those students do is fundamentally different from the day-to-day work done by historians and scientists). By comparison, junior high students playing in an orchestra, writing short stories or solving math problems are almost entirely focused on processes and those processes are essentially the same as those engaged in by professional musicians, writers and mathematicians.

Unlike music and writing, however, mathematics starts out as a convergent process. It doesn't stay that way. By the time a student gets to upper level math courses like real analysis or applied subjects like statistics ** there are any number of valid proofs for theorems and approaches to problems, but for anything before, say, differential equations, most math problems have only one solution and students are able to quickly and accurately check their work. Compare this to writing. There is no quick or accurate way to gauge the quality of a piece of prose or, worse yet, verse. Writers spend years refining their editing skills and even then they still generally seek out other critics to help them assess their own work.

This unique position of mathematics allows for any number of easy and effective self-study techniques. One of the simplest is the approach that got me through a linear algebra section taught by the worst college level instructor I have ever encountered (and that, my friends, covers some territory).

All you need is a textbook and a few sheets of scratch paper. You cover everything below the paragraph you're reading with the sheet of paper. When you get to an example, leave the solution covered and try the problem. After you've finished check your work. If you got it right you continue working your way through the section. If you got it wrong, you have a few choices. If you feel you basically understood the solution and see where you made your mistake, you might simply want to go on; if you're not quite sure about some of the steps in the solution, you should probably go back to the beginning of the section; if you're really lost, you should go back to the preceding section and/or the previous sections that introduced the concepts you're having trouble with.

Once you've worked through all the examples, start on the odd numbered problems and check your answers as you go. If you're feeling confident, you can skip to the difficult problems but if you make a mistake or get stuck you should probably go back to number 1.

Don't get me wrong. I'm not saying this is the only technique, let alone the best, for teaching yourself mathematics. Nor am I suggesting that we make a practice of dumping student in sink-or-swim situations. I think we should provide students with the best teachers and support system possible, but even under those conditions, the internal resources needed to teach yourself mathematics are tremendously valuable to all students and are absolutely essential to anyone who has to use sophisticated analytical reasoning.

Tragic postscript: In what I can only assume is an idiotic attempt raise standards, most books have stopped giving answers to odd-numbered problems. Under the old system you would assign odd problems when you wanted the students to be able to check their work and even problems when you wanted to make sure they weren't just looking answers up in the back of the book. It was a simple, flexible, and effective system that encouraged students to be independent and resourceful. No wonder it was such a prime target for reform.

* I heard a story (possibly apocryphal) of a professor who walked into an upper level math class, wrote properties of real numbers on the board, told the class that was all they needed to know to prove all the theorems in the book, then walked out.

** The relationship between mathematics and statistics is particularly complex, far too complex to discuss in a blog post.

Marginal Revolution has amazing comments

This has led me to quote a few of the best points made in the latest discussion on tenure. From PQuincy:

By the way, I do admit that tenure (which I, like Tyler, possess), may lead to the payment of rents, and to some rent-seeking behavior. But if anyone claims that, say, the modern financial sector involves no rents and rent-seeking, I'll laugh out loud. The better question is to ask what the negative burden of rents in universities is compared to potential benefits of a system that awards some kind of tenure. Given that few university professors make very much money, my first guess (no doubt shaped by self-interest) is that tenure is a relatively inexpensive way to achieve certain hard-to-value-but-real outcomes.

This does make a point that is often absent from the debate-- when we say that tenure is a form of rent-seeking is it really the current highest priority? After all, financial services are able to create all sorts of captive income streams (think of fees on 401(k) accounts) that are both more expensive and harder to justify.

From Cliff Bekar

Evaluating a good hire into an academic department is expensive. It would be very expensive for managers (Deans, Provosts, etc.) to evaluate the contributions of an academic to a literature. Hiring an outside firm would also prove a challenge given the huge variance in academic performance and the length of time it takes to determine whether a good hire has been made (it can often take 6+ years for agents to fully realize their research potential). Academics within a discipline have a comparative advantage in making such determinations (as an economic historian I would be at a significant disadvantage in evaluating, say, a new econometrician). Further, since the quality of their department stands to gain the most from making a good hire, academics within the department have an incentive to maximize the quality of hire. But imagine that the people in charge of the hire may be fired. In practice decisions to fire an academic will always be made in part relative to the standards of a department/institution. An agent that is performing poorly relative to their colleagues is more likely to be let go. A 50-year old academic, who might lose their job if faced with high performing colleagues faces perverse incentives in the hiring process. Tenure renders the agents who are best able to evaluate talent with the good incentives to hire the best possible fit. This effect is even more important if we consider other elements of an academic hire that make "insiders" better at evaluating potential fits: What types of courses would the existing student mix best benefit from, what comparative weaknesses in the curriculum should be addressed with a new hire, what is the culture of the department? Good answers to these questions are often hard to answer by simply accumulating the kind of data a third party (recruiting firm or Dean) could cheaply secure.

This is a fantastic point and really shows how tenure could work to increase excellence. Now, it is true that this argument worked a lot better with a mandatory retirement age (as end of career issues are proving sticky) but that seems like a possible point of intervention, doesn't it? Making tenure no longer automatically protect jobs after the eligibility age for social security would be the example of a small tweak that radically improves the system.

Finally:

No institution is perfect; tenure is no exception. But it is a way to allign faculty interests with the long term interests of their institutions. In this era of 5-year-turnover-administrators, without tenure what makes anyone think that short term administrators will do what is in the long term interest of their current institutions?

It's hard to align interests effectively given short term windows (industry constantly grapples with this issue) and tenure seems like a cheap way to do it in a very strong way.

It's not an accident that Marginal Revolutions was the blog that first got me to read blogs. Very insightful stuff.

Seyward Darby -- now not even pretending to think things through

The idea that the proposed educational reforms will magically fix everything has become so ingrained in Darby's thinking she has started writing things like this:

Will states, school districts, and unions also agree to revise, in the long term, the ways teachers are hired, compensated, and fired, hopefully precluding the need for mass layoffs in the future?

Even if there weren't serious problems with the proposed metrics for compensating and firing teachers, even if the changes were to do everything the advocates have claimed, how would these changes have any effect on the pro-cyclic economic policies of states? Has anyone even suggested a relationship here?

And if we're not limited to cases where there is some sort of likely causal relationship, why limit ourselves to lay-offs? Why not have something like this: "Will states, school districts, and unions also agree to revise, in the long term, the ways teachers are hired, compensated, and fired, hopefully ending future violence in the Middle East, curing cancer and preventing future movie version of SNL sketches?"?

What the hell, dream big.

A quick note on tenure and the economics of deferred compensation

(Joseph may be through with this, but I'm just getting started.)

One of the many weird things about the exceedingly weird tenure debate is the way many (hell, most) of the participants talk about tenure as a gift to teachers rather than a form of compensation that's part of a highly competitive market. *

All forms of deferred compensation offer two attractive features to employers: they reduce the initial cost and they encourage retention. Both of these features are particularly attractive to school administrators who work in an incredibly labor-intensive industry, can't offer big salaries, often require employees to move to out-of-the-way towns and can't afford to have their best people move on the moment a better offer comes along.

If you eliminate tenure you will have to come up with an alternative compensation plan that accomplishes the same thing and with the possible exception of Tyler Cowen, no one seems to show any interest in what that alternative might be.




* Of course you can also talk about tenure as a way of protecting academic freedom and discouraging grade inflation, though no one does anymore.

One last post on Tenure

I am not surprised that Tyler Cowen is asking the hard question that is at the center of the tenure debate. Namely, what is the plan forward if we replace tenure.

In particular, he asks for a concrete number for how the wage equilibrium shifts if tenure is removed:

If firing is in order, how much higher do initial wage offers have to be?

Megan McArdle gave a "toy example" that seemed more rhetorical than actual. In point of fact, moving to a small university town as a highly educated professional might be a major decision. True, in some disciplines there is a real case for a criminal over-production of PhD-level credentials. But it seems odd that the solution to this problem is to make the one part of the job chain that is actually good into something that is worse.

I'm always delighted by Tyler Cowen's insights and recommend reading the whole thing.

Paul Krugman and the limitations of historical analysis

I really like this post from Paul Krugman both for the reasoning and, more importantly, his awareness of the limitations.

So, about the first point: economic history is a great source of evidence about how the economy works – in fact, pretty much our only source. And the RR project, drawing on evidence from much further back and farther afield than usual among economists, was a great idea.

But there are usually major problems with historical analysis, no matter how much data you have, because it’s very difficult to isolate the things you’re interested in. There’s an old line to the effect that everything in the economy affects everything else, in at least two ways; this gives enormous room for spurious correlation. Econometrics is supposed to provide ways to disentangle the effects of multiple factors, but it’s difficult, and my sense is that few big arguments in economics have ever been settled by multiple regression analyses, let alone by all the sophisticated techniques developed these past three generations.

But Reinhart-Rogoff is relatively robust to these problems. Why? Because it focuses on extreme events. Financial crises are very big things, sharply concentrated in time. As a result, it’s reasonably certain that the economic developments in the aftermath of a financial crisis were driven by that crisis, not by other stuff that may have been going on.

Of course, not all economists are so careful when it comes to drawing their conclusions.

More on Tenure

I usually really like DeadDad. But this line in his post today:

It’s about recognizing that the tenure system feeds the adjunct system, and that the only way to get rid of the latter is to get rid of the former.

seemed to strike a really odd note. It seems to be assuming that if you removed job security (aka tenure) from the people with benefits then you would see adjuncts being treated better. This statement assumes a couple of things. One, is that there is a pile of resources locked up by older professors who refuse to work or retire. Two, it assumes that (in no sense) can we consider a gradual fading out of the job to be a form of delayed compensation for the early years of fighting for tenure. I am not sure what it is about deferred compensation that makes people want to break agreements about it so badly but it's certainly a new epidemic.

Finally, he assumes that a surplus generated by reducing the costs of tenure would not be eaten up by: a) administration, b) higher compensation to star professors to account for lower job security or c) reduced funding to the university.

One of the things that tenure works against is the principal agent problem of state agencies: it is hard for a government to regulate bad management without the appearance of corruption. Minimizing the changes that can be done under a single term as a senior administrator (also seen in the civil service) would seem to be a partial solution.

"The Double-blind Job"

I'm not much of a Leverage fan -- if you're in the mood for a caper show I'd stick with Burn Notice and Mission Impossible (the series, not the loud and largely brainless movies) and if you want to see more Gina Bellman there's Blackeyes, Coupling and Jekyll -- but between the title and the setting of the episode below, I just had to post this.

The ASA was out of its mind to give Felix Salmon a reporting award

Is the American Statistical Association even vaguely familiar with the internet? Don't they understand how the blogosphere food chain works? One blogger says something mind-numbingly stupid which causes a wave of outraged and mocking posts in response which causes yet another wave of defences and accusations of hypocrisy.

Obviously the award should have gone to someone like Ben Stein (the phytoplankton of the blogosphere). Stein's writing is filled with the kind of nutrient-rich raw material that bloggers (including Salmon) can write about endlessly.

By comparison, bloggers trying to subsist on Salmon posts are usually left writing things like "Felix Salmon has an interesting analysis of..." or "Salmon makes an important point about..." or "Check out..."

On a related note:

Check out these very amusing posts by Felix Salmon: the first looks at the MBA-Tourette's that afflicts so many online editors; the second has fun with the aforementioned Ben Stein's claim the unemployed deserve to be out of work.

Salmon also has a devastating analysis of CNBC's muddled take on counterfeiting.

And while you're there, take a look at this post that makes an important point about the nomination of Elizabeth Warren for head of the CFPB.

Hell, just read the whole damned blog.

Bad public policy

I want to put together this post by John D Cook:

We can’t use common sense because it doesn’t fit on a form.

We can’t treat people like people because that doesn’t scale well.

We can’t use a simple approach to solve the problem in front of us unless the same approach would also work on a problem 100x larger that we may never have.

If the smart thing to do doesn’t scale, maybe we shouldn’t scale.

with this post by Radley Balko about a note sent home because a child was too heavy:

•Back when I wrote about the obesity debate for Cato, I remember when public schools telling parents their kids are too fat was the sort of thing people on my side of the debate warned about, and people on the other side of the debate said was ridiculous hyperbole. Also, the kid’s BMI is 19.4, and the school is sending home fat warnings? Why not just go ahead and build a vomitorium next to the girls bathroom?

Now, it is true that the person in question was in the 85th percentile for her group. But let us be clear here: if every single person in America had a healthy weight then somebody would still be in the 85th percentile. By definition.

We don't talk epidemiology enough on this blog (given the title) so this is a good tiem to talk about the need for careful developed public health policies. All that badly designed policies like this will do is to make it harder to implement effective and useful interventions.

To link this back to the quote at the beginning, blanket policies like this need to include nuance or else they are at risk of doing more harm than good. If we can't design a policy that can avoid these kinds of absurd outcomes then we should really be thinking about whether the area is a good one to implement policies in. But bad policies are worse than no policies . . .

This is exactly right

Candid Engineer has a very nice post about focusing on the things in your control. Luck is a major part of success but luck definitely favors the prepared. In the long run, it pays to focus on what you can control and let things onfold as they may.

New blog alert

David Warsh highly recommends this new blog by journalist Bill Bulkeley. I don't agree with his take on these companion pieces from the New York Times and the Wall Street Journal about Teach for America (time permitting I'll elaborate later), but he makes some good points and asks right question.

Definitely a site we'll want to keep an eye on.

The yahoos at Yahoo

A few days ago, the Yahoo homepage featured a Bankrate.com article that consisted of an amazing assortment of bad advice on homes as investments. I posted a rebuttal, but I could have simply waited for a few days and linked to this article from U.S. News which was also featured on the Yahoo homepage and which explains why everything the earlier piece said about home ownership was wrong.

I realize that Yahoo's homepage is not exactly a publication, but it does have tremendous reach and it would be nice if they hired someone to actually read what they post.

Divergent and Convergent Learning

As much as I complained about them at the time, the education classes I had to take to get certified did have some highly useful concepts. One of those was the distinction between convergent learning (where you want all students to reach the same final answer) and divergent learning (where you want each student to come up with a unique answer). Before you made a lesson plan or write a test, you were supposed to ask yourself where you want to see convergence and where you want to see divergence.

[There is an obvious connection between this divergent learning and the creativity discussion here, here, here and here]

There is an common but fatally naive misconception that convergent learning goes with math and science while divergent learning goes with arts and humanities. Almost all subjects start with a large convergent learning component including the arts (try picking up an instrument and see how much divergence your teacher tolerates in the first few lessons). More importantly, ALL subjects are fundamentally divergent at a high enough level. Writing a novel, composing a symphony, proving a conjecture or designing an aircraft are all creative exercises in constrained problem solving. We demand that certain conditions be met but we expect that each solution (or at least the method behind it) will be unique.

Which begs the disturbing question: will the proposed educational reforms produce more or fewer of the divergent thinkers we actually need?



p.s. Liam Hudson came up with a uses of objects test to measure divergent thinking. It ignores the complex interaction between possibility and constraint and is therefore, in my ever-humble opinion, complete crap, but take a look anyway.

Bias versus precision

In epidemiology, we are typically trying to estimate an unbiased measure of associations between an exposure and an outcome. Generally, we punt on the causal question of "does exposure X cause outcome Y?", but it is inevitably in the background. After all, if we say that poor exercise habits are associated with early mortality it is generally taken as an advisory to consider improving one's exercise habits rather than as an interesting coincidence.

But not all models are confounding models and the instincts that serve us so well for confounding models can be misleading for predictive models. Nate Silver has a very well explained example of how inaccurate (or, to be more formal, imprecise) predictive models can be worse than biased models. It's a very interesting confusion between bias and precision but it makes me wonder if we don't focus too much on unbiased and too little on efficiency for some of our models.

Thoughts?

More on creativity and slippery metrics

John D Cook has a interesting take on the whole creativity question. I'm not sure that I accept the premise that schools have as much influence on these factors as people think. But it does bring up an interesting point of the tail wagging the dog.

On of the classic criticism of scientific management is that they took a good idea (try and find measureable metrics of business success and failure) and turned it into a myopic focus on what is easy to measure in a spreadsheet. As a result, they focused on easy to measure metrics of success and, in the process, tended to neglect things that are hard to measure.

It's obviously true that creativity is a slippery and hard to measure concept. It is hard to, for example, design a standardized test around it. I wonder if part of the issues of education reform come from a focus on what can be measured and not what is important?

The best way to get rich is to sell books to gullible people

I sometimes wonder how much the current crisis was acerbated by bad advice from financial gurus on TV, advice that has changed amazingly little. Here's a recent example from David Bach (author of Start Late, Finish Rich and endless variations):

The best type of debt is debt that builds wealth over the long run, and the
No. 1 example of that is mortgage debt.

"Home values have increased an average of 6.5 percent a year over the past 30 years," says Bach. "So when you borrow to buy a home, chances are that's good debt. You'll build value."

Bach heavily promotes the idea of homeownership, saying that everyone needs to own where they live. "About 40 percent of Americans are renters," says Bach, "and the fastest way to wealth in America is buying where you live."

Bach cites some shocking numbers to back this up. "The average renter has a median net worth of $4,000, and the average homeowner has a median net worth of about $150,000."

...

One of the reasons so many Americans seem mired in bad debt (Bach reports that the average American carries approximately $8,400 in credit card debt) is that financial education is pratically (sic) nonexistent. "This type of commonsense stuff isn't taught in school," says Bach, "and most Americans don't realize how bad high-rate credit cards are hurting them."

Bach has been doling out this same advice complete with the same net worth statistic (apparently included under the assumption that schools didn't teach statistics either) for about a decade*. It is slightly more sound now than it was five years ago, but it is still not what you'd call good.

This ground has been covered before (notably by Felix Salmon) but just to recap:

1. Home ownership is not a particularly good investment either in terms of returns. It's bad in terms of liquidity and horrible in terms of diversification;

2. Analyses purporting to show the amount saved by owning vs. renting almost always overlook the direct and indirect costs associated with suburban commuting.

3. Home ownership makes it difficult for workers to go where the jobs are. This has economic consequences on the national level, but it's on the individual level that this can turn into a real horror show. Housing prices, liquidity and employment are all correlated by region.




*In Start Late, Finish Rich, Bach says "you cannot get rich being a renter."

Creativity

Felix Salmon has a post on creativity that got me thinking about comparisons. After all, in 1966 (when the test that he discusses was developed), it was a lot harder to get access to first rate creative materials. In that sense, "do it yourself" made a lot of sense and might have encouraged creativity.

You can see this as technology progresses. Consider live music and plays over time. In 1890, if you wanted to hear music or see a story acted out in your home town then you needed to get a local musician to play or go to a local community theater. In 2010 you downloaded the music to your iPod or a top film to your television set.

You see this phenomenon with lots of things. When my dad did weight lifting in the 1960's, the best you could do was a book on the topic. People competed to invent new techniques (and likely had a lot of injuries as a result). Today, I can find massive amounts of information on the topic sitting at home. In the 1970's, Gary Gygax and company could create a new type of game (role-playing games) as an extension of war gaming. Today the ability to create a new game is much harder given the development costs.

So is the lack of creativity a symptom of a richer environment where finding rhea optimal solution is better than creating a new (and likely sub-optimal one)?

I am not saying that this is the case and Felix Salmon brings up a lot of other good points. But it's another reason to wonder about what the implications of these claims would be, even if they were to prove to be correct.

Net impact

The ever interesting Professor in Training has a post on the issues of extraneous demands on a professor's time. I think that this is another example of things where each individual element is likely to be justified but the whole is not.

Examples of these sorts of things include regulations, old laws (regarding horse thievery, for example) assistant professor committee work, software patches and so forth. For example, the last university I was at required a one hour course on the hazards of asbestos every year. While I am sure that asbestos is deadly substance, it seems like overkill to have a class on such a narrow topic every year.

What is hard to determine is how to identify these situations ahead of time and determine the appropriate responses. It's not easy -- as a junior professor I have said no to a few requests already that were pretty reasonable (and sounded like they'd add value) just to make sure that I had enough time.

I would like to think of a way to handle this issue that did not involve "over-all impact committees".

Another reason not to aggregate elementary and post-elementary school data

This is from a National Center for Education Statistics summary of TIMSS data from the Nineties* when the reform movement was pretty much settling into its present form. They were already sounding the alarm about math and science scores and proposing extensive reforms for grades K through 12, despite the fact that the performance issues were almost entirely limited to junior high and high school.

In mathematics, fourth-grade students in 7 countries outperform our fourth graders (Singapore, Korea, Japan, Hong Kong, the Netherlands, the Czech Republic, and Austria). Students in 6 countries are not significantly different from ours (Slovenia, Ireland, Hungary, Australia, Canada, and Israel). U.S. fourth graders outperform their counterparts in 12 nations (Latvia, Scotland, England, Cyprus, Norway, New Zealand, Greece, Thailand, Portugal, Iceland, Islamic Republic of Iran, and Kuwait).

In science, students in only one country--Korea--outperform U.S. fourth graders. Students in 5 countries are not significantly different than ours (Japan, Austria, Australia, the Netherlands, and the Czech Republic), and U.S. fourth graders outperform their counterparts in 19 nations (England, Canada, Singapore, Slovenia, Ireland, Scotland, Hong Kong, Hungary, New Zealand, Norway, Latvia, Israel, Iceland, Greece, Portugal, Cyprus, Thailand, Islamic Republic of Iran, and Kuwait).

*I'll try to dig up some more recent results just to see how things are trending.

If all students were 15 years old...

Joseph has an interesting link to a comparison of test scores of fifteen-year-olds in different countries. I recommend you go by and take a look. It has some interesting stuff, but what caught my eye was what wasn't there. I couldn't find a complementary break-down for any other age group.

After bad metrics, I think the worst problem in educational research may be the inappropriate aggregation of primary and secondary education. If there was ever a case where two populations should be treated separately it's here. The two systems face different problems of different severity that respond to different solutions.

I'll try to follow up with some specifics later, but in the meantime, when you hear someone making a blanket proclamation about our schools, remember that about the only meaningful statement you can make that's true about primary and secondary schools is that the teachers in both are about to get screwed.

Statistical programming languages

This post by John D Cook got me thinking about whether it is possible to do a similar simplification of programming languages for Epidemiologists doing analysis.

These days I see the following languages in heavy use: STATA, R, SPSS, SAS and some S-plus. Furthermore, one is requires to do data management in some combination of SQL/Oracle, SAS, Excel and Access. That doesn't even touch on the people who still use C++ and/or FORTRAN for specialized programming applications.

My question is which ones does it make sense to support? In my department, I think we'll do a combination of R (freeware, flexible, powerful) and SAS (FDA standard) as languages that we officially support. But not supporting STATA is a very painful choice! What have others done in similar circumstances?

Speaking for the unhinged

Jonathan Chait dismisses critics of proposed education reform as 'unhinged.' Speaking as one of the whackjobs, here's the sort of thing that makes us loonies so nervous.

From a press release (6/8/09) from the Department of Education:

Emphasizing the need for additional effective education entrepreneurs to join the work of reforming America's lowest performing public schools, U.S. Secretary of Education Arne Duncan told reporters during a conference call this afternoon that states must be open to charter schools. Too much is at stake for states financially and for students academically to restrict choice and innovation.

"States that do not have public charter laws or put artificial caps on the growth of charter schools will jeopardize their applications under the Race to the Top Fund," Secretary Duncan said. "To be clear, this administration is not looking to open unregulated and unaccountable schools. We want real autonomy for charters combined with a rigorous authorization process and high performance standards."

From a May 1st story in the New York Times:

But for all their support and cultural cachet, the majority of the 5,000 or so charter schools nationwide appear to be no better, and in many cases worse, than local public schools when measured by achievement on standardized tests, according to experts citing years of research. Last year one of the most comprehensive studies, by researchers from Stanford University, found that fewer than one-fifth of charter schools nationally offered a better education than comparable local schools, almost half offered an equivalent education and more than a third, 37 percent, were “significantly worse.”

Us nutjobs would like to see the administration reconsider its position based on the data.

"Fewer Low-Income Students Going to College"

From the Wall Street Journal blog Real Time Economics (via Thoma):

Fewer low- and moderate-income high school graduates are attending college in America, and fewer are graduating.

Enrollment in four-year colleges was 40% in 2004 for low-income students, down from 54% in 1992, and 53% in 2004 for moderate-income students, down from 59% over the same period, according to a report recently submitted to Congress by the Advisory Committee on Student Financial Assistance.

If that trend has continued, low- and moderate-income students who don’t move on to college face an even darker outlook. The unemployment rate for 16- to 19-year olds averaged 17% in 2004, the jobless rate for people over age 25 with just a high school diploma averaged 5% the same year. So far this year, those figures have jumped to 25.8% and 10.6%, respectively.

College expenses and financial aid have become increasingly larger considerations for parents and students, driving more qualified students away from enrolling in four-year colleges.

Harm Reduction

We've been a little bit into economics and education lately, so I suppose here is a good chance to look a publich health issue of interest. In today's British Medical Journal, Kimber et al. report on a prospective cohort study looking at opiate substitution in injectable drug users.

The findings were dramatic:

For each additional year of opiate substitution treatment the hazard of death before long term cessation fell 13% (95% confidence interval 17% to 9%) after adjustment for HIV, sex, calendar period, age at first injection, and history of prison and overdose.

This is a massive effect. Just one year of treatment is competitive with interventions that we would not even think twice about: many drug treatments for cardiovascular disease fall into this range. Since we can't really randomize people to these types of treatments, the prospective cohort design is likely as good as it gets. So perhaps these types of dramatic findings, if properly replicated, can start a discussion about harm reduction once the behavior is initiated. After all, there is a large population of injectable drug users and finding ways to improve their (fairly poor) outcomes seems like a decent way to improve public health.

More on tenure

In a thread on DrugMonkey, a commenter posed this solution to tenure:

The society as a whole would benefit from abolishing tenure system and putting educators where everyone else belongs, a market place.

As you may guess, I had some thoughts about this comment. Markets are pretty amazing things but they do some things rather poorly. Mark Thoma has a good discussion of the criteria for a market to work efficiently:

http://economistsview.typepad.com/economistsview/2007/02/markets_are_not.html

Research is, in some important respects, a public good and, like other public goods is hard to place into a market easily (in the same sense that is hard to privatize the road system). Furthermore, the NIH style grant system is an attempt (so far as I can tell) to handle the information problems with basic research in the most rational way possible under the circumstances.

This is not to say that change and reform are not possible; they are. But I find it odd that tenure is attacked whereas other forms of job security are not (for example, there is no widespread call to privatize the military). The university and NIH system are designed to introduce as mush competition into research and teaching as you can with a single buyer dominating the marketplace (that would be the federal government, via student loans and research grants).

I think tackling this requires a series of marginal improvements. Standing on the outside it is easy to suggest radical change but what do you do if the radical change results in a worse outcome than the current state of affairs? In particular, I want to see more about how to apply a market based system to higher education given all of the barriers to an efficient market that are present.

Post tenure era?

DrugMonkey has a post today called "A Farewell to Tenure". It's an interesting post in the context of Mark's discussion about the ending of teacher tenure in the K-12 system.

However, I think that the major driver of the reason that tenure can be attacked in the academic world is that there is a dual career track for many academic disciplines. The fields with lots of soft money jobs (employment based 100% on external grant funding) tend to be focused on research topics that are seen as priorities. Here there are specific metrics (such as key scientific papers published and novel ideas proposed) that make it possible to actually hold competitions for funding.

What is the analogous parallel funding track for K-12 teachers?

Thinking about teacher layoffs -- the logistics of staffing

Regarding Seyward Darby's recent article on teacher layoffs, you can find posts on why she doesn't understand economics here and here, posts on the questionable data and analysis behind her position here and here, posts pointing out the problems with the movement she endorses here and here (from Darby's own magazine) and an explanation of why the tenure 'reform' she champions would be unfair and unethical here.

But I don't believe anyone has pointed out what a logistic nightmare she wants to get us into.

Let's take the plan that Darby favors and throw in some reasonable numbers:

We take our already somewhat understaffed schools and lay off, let's say, 200,000* teachers selected based in part on some quality metric. We then run the schools severely understaffed for the next year or so until state revenues recover.

That 200K will be made up of three groups, the good, the bad and the better-than-nothing. The good are effective teachers who end up on the list through a combination of bad luck, bad metrics and bad administrators (go here to see how the last two can work together). The bad are teachers at the very bottom of the quality scale. The better-than-nothing are teachers who aren't all that effective but are probably still at least as good as most of the people you could get to replace them.

Given the flaws in our system of ranking teachers and the innate difficulty of the problem, a fair number of good teachers will end up on the list. Likewise, given the problems with recruiting teachers for problem schools, better-than-nothing teachers will continue to make up a large part of it.

The upshot of all this is that, given population growth and the high level of teacher attrition, you will need a substantial number of the people you laid off to come back if you are to have any hope of meeting staffing targets. This wasn't as much of a problem under last-in/first-out for a couple of reasons. First, the attrition rate for new teachers was so high that many of the teachers you laid off wouldn't have been there in a couple of years even if you hadn't let them go. Second, leaving under last-in/first-out carries minimal stigma. For those who really wanted to it was easy to get back into the field a year later.

Under the 'reform' system, there is a serious stigma and a deadly asymmetry of information problem. Keep in mind that administrators are basically stuck with new hires for a year (you try finding a certified replacement in, say, November). They know that a teacher laid off under that system might turn out to be first rate, but do they really want to take the chance?

Let's say 100,000 of the laid-off teachers fell into the need-them-later category. We have screwed these teachers out of contractually obligated compensation, scapegoated them for all the problems in the education system and made them unemployable in their chosen field. Would you come back under those circumstances?

Of course, the impact is not limited those laid off. Other than the satisfaction that comes with the job, the primary attraction of teaching is job security. For generations, teachers have self-selected to be risk-averse. Faculty rolls mainly consist of people who have willingly traded any significant chance of promotion for the stability to start a family, buy a house, and spend thirty or forty years doing something they're good at.

In an industry dependent on a very large number of highly skilled labourers, this is not a bad situation, but putting those bigger, long-term questions aside, what is likely to happen when the economy recovers and we try to start restaffing? We were having trouble maintaining adequate levels before the lay-offs. Then we permanently removed around 200,000 teachers from the pool (many of whom we would have liked to have had back later). Now we are trying to hold on to teachers after having unilaterally removed a valued part of their compensation and established the precedent that contractual obligations to educators don't need to be honored.

Of course, the reformers will assure you that they have a set of programs that will solve the retention problem, but keep in mind these are the same people who have been solving this problem for a decade and their track record is nothing if not consistent.

According to this NCES paper, we were hiring around 200,000 new teachers a year before the economy collapsed. That was a difficult and expensive proposition and we often had to compromise quality to make that number. Now think about doubling or tripling the number of vacancies.

This is not going to work out well.







*without the plan that Darby thinks might merit a veto, 100,000 to 300,000 teachers could lose their jobs.

Wayward Seyward

I mentioned in my previous post that when I saw Seyward Darby had written an article entitled "Obama: Don't Touch Education Reform," I had a bad feeling. Bad and, as it turns out, damned accurate:

But this particular measure has evolved in some decidedly unappealing ways. When it was first introduced, numerous education advocacy groups asked Congress to impose constraints that would effectively compel states to reform their layoff systems. As I explained recently, most states and school districts follow a misguided "last-hired, first-fired" rule. If they must let teachers go for budgetary reasons, they start by booting those teachers who have spent the least time in the classroom. Teacher quality doesn't factor into the decision at all. Congress, though, refused to go along with the proposal—and the Obama administration didn't intervene. "Right now, the most important thing is to stop the bleeding," Senator Tom Harkin, chair of the Health, Education, Labor, and Pensions Committee, said in early May.

That was bad enough. But then, this week, House Appropriations Chairman David Obey proposed that the government pay for the legislation, in part, by taking money from other Education Department funds—and not just any old funds, but the money set aside for some of Obama’s most important school reforms. Under Obey's plan, which the full House ultimately adopted, $500 million would come from Race to the Top, a competitive grant program and probably the most talked-about aspect of Obama's education agenda; $200 million would come from the Teacher Incentive Fund, which supports performance-based compensation plans; and another $100 million would come from money for charter schools. Sounding a bit like Harkin, Obey reportedly said, "When a ship is sinking, you don't worry about redesigning a room, you worry about keeping it afloat."

...

And if the push for other funding fails? The administration will have a tough choice to make. But one thing's for sure: Letting Congress chip away at the education reform agenda now would place it on a slippery slope in the future. And that's something the country can ill afford.

"Tough choice"? Let's review:

1. We are on the verge of adding a Roman numeral to the end of the Great Depression. Under these circumstances, mass layoffs of teachers is insanely pro-cyclic. It doesn't matter which teachers you pick, the immediate economic effect is the same. Even if you had a reliable metric that could identify poor teachers, this would be the worst possible time to get rid of them;

2. You don't have a reliable metric that can identify poor teachers;

3. Much of the threatened reform package is actually a continuation of policies that have a long (sometimes decades-long) and horrible track record.

Narrow versus broad searches

The ever interesting Felix Salmon has a post today on hedgehogs versus foxes that I found quite interesting. In it, he makes a distinction between a careful and incremental approach to knowledge (traditional research) versus a fast and haphazard approach (blogging). I find this especially fascinating because I tend to be an incrementalist as a research (let's refine the current optima) whereas I am much more willing to discuss things for which I am not always an expert on a blog.

What struck me as most interesting is that a complete strategy for improving a field likely embraces both approaches. The bloggers (foxes) are racing around asking naive questions in the search for an undiscovered solution. The academic researchers (hedgehogs) are refining the known solutions to make them actually optimal. Both approaches are actually very useful, on their own terms.

The thing to keep in mind is that the fox approach is going to generate a lot of failure. When I do epidemiology methods research, I always keep in mind that just beating he standard practice in the field is very hard to do. Heck, for a lot of problems, just getting substantial improvement in estimation over a carefully thought out linear regression model (regardless of whether the outcome is continuous or not) is surprisingly challenging. It can certainly be done and for some classes of problems, like time to event, the improvement is dramatic. Even more difficult, the current tools have been carefully refined and well understood -- developing a superior alternative is hard (just look at the number of methods papers published every month in statistics in medicine).

But every once in a while, a new idea shows up that leads to dramatic improvements, one way or the other, over conventional approaches. I think bloggers do a lot of the grunt work of looking around and trying to see if there is a superior paradigm or approach to looking at the current problems. Then academics take the slow and difficult process of incremental improvement.

It's a nice way of looking at the relation between these two approaches.

Risk Taking and success

Tyler Cowen, on his ever excellent blog, quotes a reader on how people take chances in some areas of their lives and are quite bland in others. I found this interesting at several levels. First, I tend to agree with the general principle that we have only so much energy (mental or physical) to spread between tasks and that you can be focused on only so many at a time. Second, I think it blends in well with Mark's recent skepticism that risk taking is as good of a universal strategy as it seems to be if you read business advice books.

The reason is that success is not a single dimension, nor is happiness. If you have the incredible natural talent and drive to be the very best you can possibly make a risk taking strategy work because you have the ability to recover from failure. In the same sense, the very rich can try high risk investment strategies as they may still be well off even if the strategy fails. My favorite example of this is somebody like Garth Brooks trying to be Chris Gaines; Garth was a sufficiently well known star that he could recover when the "Chris Gaines gamble" did not work out.

But to reach these levels of focus, is it not likely that other things in life may have to be sacrificed? It is true that highly successful people are driven and focused but do need to remember that this focus often comes at a surprising cost. Alexander the Great was a successful general but would anybody envy his personal life?

All of which is to say that strategies for success are complicated, at best, and that people aren't simple at all. Which, come to think of it, is probably a good starting point for all of these discussions.

Uh-oh

Seyward Darby is writing about education reform again. This hardly ever ends well.

Role models

While browsing the business section yesterday, the title and author of one of the books caught my eye so I picked it up and scanned the back cover:

Inspiration. Success. Confidence. Passion.

No one is born with these qualities, but they are the key ingredients for reaching goals, building careers or turning a blueprint into a breathtaking skyscraper.

I learned that the author would share "the life lessons and hard-won insights that made her a rising star in the business world" and that whether the reader was "landing that first job, navigating the workplace,or making a lasting impact," the author would show how to "Step up and get noticed at work -- focus and efficiency will open doors."

The author's name was Ivanka Trump.

The book is the Trump Card.

It's available in paperback.

How blogging is like SCTV

The creators of the cult classic, Second City Television had a admirable approach to editing. They stopped when they ran out of things to say. If that meant slight gems like "Western Redundancy Playhouse Theatre" or "The Wacky World of Poverty" only ran a couple of minutes then that's how long they ran. The writers also weren't afraid to keep going as long as a premise held out. If they had an hour's worth of ideas for a Poltergeist parody, they would give it an hour.

It is rare in any medium to be free make things the length they ought to be, but a blog is one of the few exceptions. In a blog you you spend hundreds of thousands of words on a subject or you can cover it in the span of a haiku. We don't always make the best use of that freedom but we should at least appreciate having it.

An actress prepares for the wrong role -- 3:06


Dick Cavett finds the perfect guest -- 2:44



A schlocky horror show accidentally books an art film -- 8:11

Paper review times

Disgruntled PhD has a nice post about an issue for early career scientists:

Another point to remember is that you can't submit to multiple journals at once, so the reviewing time is an opportunity cost for the researcher. This is of particular relevance for students like myself, who need to get papers published quickly in order to be able to show them on a CV and thus get a job (and the opportunity to do more research.

Later on, when one has a track record, it makes sense to aim high on a paper that has potential. It's never possible to be sure about a specific paper as one of the factors that goes into the decision of a journal to accept or reject a paper is what other papers have arrived recently. Back when I started out in Epidemiology (seems like a lifetime ago), this was true of our journals (like the American Journal of Epidemiology) as well. Fortunately, AJE has made a massive effort to improve things. This paper by Raymond J. Carroll shows a review time for AJE in 1999 of 15 months (figure 3). Yikes!

On the other hand, it is unclear what the optimal point is. I know of researchers who submit all of their papers to the top five medical journals "just in case" because it only takes a couple of months and, who knows, lightning might strike.

But it remains a very tough problem for junior people, like myself, who need to maintain momentum (both the volume of papers and the speed of papers). I do like the BMC Medicine model where the general journal reviews the paper and then refers the paper to a sub-journal if they think it would work there instead. I would publish in BMC journals a lot more if I was not so broke as I like their actual policies and practices a lot!

Mark

My co-blogger (and half owner) of this site, Mark, has a great eye for poor reasoning and likes to point it out. You can see the reaction to one of his forays here.

Charms and Mogul Dust

The recent discussion of Seth Godin (see here, here and here) got me thinking about how far business gurus will go out their way to build an association with the fabulously successful, even when that success is built on incredible talent and/or luck and could not possibly be applicable to a general audience. Godin does it with Dylan, probably the most influential songwriter of the second half of the Twentieth Century. Another popular example is Michael Jordan, quite possibly the greatest basketball player ever.

There is really no way to draw a useful analogy between the careers of Dylan and Jordan and what the rest of us face, but of course, that's not the point. The gurus use these men in their examples for the same reason that our ancestors would dress up like bears and growl and snarl at each other around the campfire. It's a primitive but still appealing magic of association and imitation.

Eating a box of Wheaties with Derek Jeter on the front will not raise your batting average, wearing a LeBron James jersey will not help your jump shot and trying a career move just because it worked for Dylan or Jordan will probably turn out to be a bad idea.

One less reason to read the New York Times

Olivia Judson is taking what is supposed to be a year's sabbatical from her column at the New York Times. You can't really begrudge her the time off; she's certainly earned it. Still it makes me nervous. Sabbaticals have a way of turning into permanent departures and that's not hit the paper can take. For a long time now, Judson has been the designated outlier, balancing out a tremendous amount of mediocrity.

We've lost Judson's subtle and entertaining insights into evolution. We're left with John Tierney asking if your dog is really sincere.

Publishing Equilibria

This comment at the end of a blog post by Andrew Gelman got me thinking:

On the other hand, the current system of scientific journals is, in many ways, a complete joke. The demand for referee reports of submitted articles is out of control, and I don't see Arxiv as a solution, as it has its own cultural biases.

I wonder if the problem here is not one of equilibria. Journal articles have become the an important way to measure productivity and to define success. As a result, researchers who are good at producing journal articles prosper. To change the system you have to overcome the intertia of the decision makers often being those reseachers who were successful at the current system.

Most systems of prestige tend to be hard to overturn and it's unclear as to what would be a good pathway to doing so. I tend to wonder about marginal improvements as these are more likely to be adopted. But it is pointless to fight for scientific expression to be done on blog posts if employers count grants and papers to determine future employment.

So what are the improvements at the margin that are possible?