Dean Dad on Zero Sum Performance

This is Joseph.

From Dean Dad:

In most states or systems with performance funding, the overall level of funding -- the pie to be sliced, if you prefer -- is either flat or declining.  Which means that if everyone improves by the same five percent, then everyone gets the same zero percent increase.  You may be making progress, but you’re still essentially running in place.  Worse, if you improve by three percent but the statewide average is five, you actually lose ground.

I find these sort of systems to be extremely tough environments to build motivation and success in, so I am glad that they are being scrutinized.  One issue is that it creates some very perverse incentives.  Consider this in a human resources context -- you do well at your job, get promoted, and now you are at the bottom of the ranking for the new rile you are in.  If the bottom group tends not to survive (long term) it suggests promotion is bad.  Or that politics will be played to make a promotion survivable, which can be pretty toxic. 

Where I have seen this system thrive is in very high reward environments.  Only one actress could be cast to play "Rey" in the new star wars film (no matter how good the second best applicant was), but there are no shortage of volunteers because the pay-off is so great. 

But placed into a less highly leveraged environment and it is a recipe for lowering motivation and, occasionally, penalizing the colleges that take on the toughest challenges.  We all want to think that we are so awesome that we can do amazing things with tough problems, and sometimes people do, but it can be a thin line between that and being set up to fail. 

A subtle issue with standardized tests

This is Joseph.

Dean Dad has a nice piece on assessment.  A part of it that jumped out was:

Johnson’s argument is subtle enough that most commenters seemed to miss it.  In a nutshell, he argues that subjecting existing instruction to the assessment cycle will, by design, change the instruction itself.  Much of the faculty resistance to assessment comes from a sense of threatened autonomy.  Johnson addresses political science specifically, noting that it’s particularly difficult to come up with content-neutral measures in a field without much internal consensus, and with factions that barely speak to each other. 

He’s right, though it may be easier to grasp the point when applied to, say, history.  There’s no single “Intro to History” that most would agree on; each class is the history of something.  The ‘something’ could be a country, a region, a technology, an idea, an art form, or any number of other things, but it has to be something specific.  Judging a historian of China on her knowledge of colonial America would be easy enough, but wouldn’t tell you much of value.  If a history department finds itself judged on “scores” based on a test of the history of colonial America, then it can either resign itself to lousy scores or teach to the test.

This means that the design of standardized test is crucially important if students and/or teachers are going to be evaluated on them.  For some subjects, e.g. basic math, this may be less controversial but it still involves making choices about what the emphasis will be.  A perfect test is like a perfect teachers -- neither beast really exists in nature. 

But this is critically important for high stakes tests, because what is taught cannot help but be influenced by the test.  If history questions on the high stakes tests are all focused on colonial America, guess what the history section of classes will look like.  In some sense that is okay, insofar as we have a broad consensus as to what should be taught.  But it does make the content of the tests a matter of public policy and concern as much as any other aspect of school instruction.

Safety Nets and Canada

Dean Dad has an interesting observation:

I was reminded of that a few days ago, in a discussion with a Canadian colleague.  We have similar senses of humor, so we got to talking about The Kids In The Hall, SCTV, and national styles of humor.  (For my money, “Brain Candy” is a neglected classic of dark, dark, dark comedy.)  She offered the theory that Canada punches above its weight culturally because its social safety net -- health care most conspicuously -- makes it possible for people to take chances on creative careers.  As a result, they get Holly Cole, and we’re left with Adam Sandler. 

That was then expanded on in the comments

While I disagree with the specific point about Canada punching above its weight culturally (quick name a great Canadian film that's not "Strange Brew"), I do think that a robust safety net does make entrepreneurial risk taking more likely because people can afford to take the risk of starting a business without having to worry about losing health insurance or other benefits.
I used to have a state government job where this dynamic was apparent: the secretaries in the agency were fairly low paid, but had very good benefits. 3/4 of the secretaries in my officer were married to husbands who had their own small contracting or (vaguely) construction related business. They made much more than their wives made, but had no independent health benefits of their own  

I think that this is a neglected conversation.  The ability to take risks is not just driven by rewards but also by the costs of failure.  If you make the rewards extreme and failure punishing then you create incentives for cheating and "doing anything to win".

This effect shows up in a number of areas -- imagine you are a high school teacher diagnosed with a major illness.  In the real world, COBRA is unaffordable and unemployment is over eight percent.  You are teaching less well due to health issues.  One can see a lot of pressure to find a way -- any way -- to keep test scores above the retention threshold.

You can also see this with small businesses.  The reforms of bankruptcy law (making it harder to go bankrupt) and the cost of health care for those without insurance makes starting up a small firm really risky.  It makes a lot more sense to stay in your sub-optimal office job with the result that you have less innovation and dynamism in the economy.

These effects are just as predictable as free markets are and it can make a lot of sense to invest in ways to pool or mitigate the risk associated with being an innovator.

Dean Dad on Higher education

While I was away, I missed this gem by Dean Dad about the argument that college isn't worth it:

The whole enterprise just smells to me like the latest variation on “let’s privatize Social Security” or “let’s replace Medicare with vouchers.” It’s the wealthy and their worshippers sloughing off any social obligation, basically dropping the ladder behind them. If that weren’t the case, if they actually believed what they said, I’d expect to see the best and brightest from Choate and Philips Exeter eschewing college and doing startups or joining the military instead. Um, no.

I had not made this connection but it does seem to be a coherent interpretation of an otherwise puzzling argument. I must admit that I remain mystified with the current interest in the United States with disassembling the social infrastructure. Not only is it in the opposite direction of most countries, but the ones that have tried it seem to end up being bad places to be. Think of Russia, for example.

The real issue, to me, is that the real remedy to these types of escalating prices is the high quality public university system that countries like Canada and states like California have. The University of California is a high quality set of institutions and much less expensive than the alternatives.

Why is this approach not the one that rising prices brings us back to? Okay, we'd have to go back to the brutal taxes of the Reagan or Clinton eras, but I am not convinced that this move would lead to an immediate dystopia.

Concerns with data driven reform

Dead Dad has a post on Achieving the Dream, which is intended to improve outcomes at community colleges. Two of his commentators had really interesting insights. Consider mathguy:

Consider the effect of No Child Left Behind. I've seen a noticeable decline in basic math skills of students of all levels in the last 5 years. Every year, I will discovered a new deficiency that was not seen from the previous years (we are talking about Calculus students not able to add fractions). Yet NCLB was assumed to be "working" since the scores were going up. It seems that K-12 was devoting too much time preparing the students for tests, at the cost of killing students' interest in math, trading quality instruction for test-taking skills. Is NCLB a factor in the study? Are socio-economic factors examined in the study?

or CC Physicist who stated:

I look at what Asst Prof wrote as an indication that a Dean, chair, and mentor didn't do a good job of getting across the history of assessment. Do you know what "Quality Improvement" program was developed a decade earlier, and what the results were of the outcomes assessment required from that round of reaffirmation of accreditation? Probably not, since we have pretty good communication at our CC but all the negative results from our plan were swept under the rug. The only indication we had that they weren't working was the silent phase-out of parts of that plan. Similarly, data that drove what we did a decade ago were not updated to see what has changed.

I think these two statements capture, very nicely, the main issue I have with the current round of educational reform. One, if you make meeting a specific metric (as a measure of on underlying goal) a high enough priority then people will focus on the metric and not the actual goal. After all, if you don’t then your name could be posted in LA Times although with your underperformance on the stated metric. So we’d better be sure that the metric that we are using is very robust in its relation to the underlying goal. In other words, that it is a very good representation of the curriculum that we want to see taught and measures the skills we want to see students acquire.

Two, trust in evidence based reform requires people to be able to believe the data. This is one area where medical research is leaps and bounds ahead of educational research. A series of small experiments are attempted (often randomized controlled trials) while the standard of care continues to be used in routine patient care. Only when the intervention shows evidence of effectiveness in the trial environment is it translated into routine care.

In education, such trials are rare indeed. Let us exclude natural experiments for the moment; if we care enough to change the education policy of a country and to violate employment contracts then it’s fair to hold ourselves to a high standard of evidence. After all, the lotteries (for example) are not a true experiment and it’s hard to be sure that the lottery itself is completely randomized.

The problem is that educational reforms look like “doing something”. But what happens if the reforms are either counterproductive or ineffective (and implanting an expensive reform that does nothing has a high opportunity cost). The people implementing the reforms are often gone in five to ten years but the teachers (at least now while they have job security) remain to clean up the wreckage afterwards.

I think that this links well to Mark's point about meta-work: it's hard to evaluate the contributions of meta-work so it may look like an administrator is doing a lot when actually they are just draining resources away from the core functions of teaching.

So when Dead Dad notes: “Apparently, a national study has found that colleges that have signed on to ATD have not seen statistically significant gains in any of the measures used to gauge success.” Why can’t we use this evidence to decide that the current set of educational reform ideas aren’t necessarily working well? Why do we take weak evidence of the decline of American education at face value and ignore strong evidence of repeated failure in the current reform fads?

Or is evidence only useful when it confirms our pre-conceptions?

Administration

I was reading Mark's post today on principals and I thought it was interesting to see that Dean Dad has a post from the other side of the fence today as well. Some of the points in the Dead Dad post are extremely insightful. Consider the following question:

I’ll answer the question with another question. Good, strong, solid, peer-reviewed scientific data has made it abundantly clear that poor eating habits lead to obesity and all manner of negative health outcomes. There’s no serious dispute that obesity is a major public health issue in the US. And yet people still overeat. Despite reams of publicity and even Presidential support for good eating and exercise habits, obesity continues to increase. Why?

In other words, reform is hard to do even when you know where you want to go. In cases where the evidence is weak or where budgets are falling, the problem gets even worse. And, of course, right now we are experiencing a fall in most education budgets in the United States.

However, it was interesting how Dead Dad was unable to resist worrying about tenure as a barrier to reform:

There’s also a fundamental issue of control. Faculties as a group are intensely protective of their absolute control of the classroom. Many hold on to the premodern notion of teaching as a craft, to be practiced and judged solely by members the guild. As with the sabermetric revolution in baseball, old habits die hard, even when the evidence against them is clear and compelling. There’s a real fear among many faculty that moving from “because I say so” to “what the numbers say” will reduce their authority, and in a certain sense, that’s true. In my estimation, this is at the root of much of the resentment against outcomes assessment.

Even where there’s a will, sometimes there just isn’t the time. It’s one thing to reinvent your teaching when you have one class or even two; it’s quite another with five. And when so many of your professors divide their time among different employers, even getting folks into the same room for workshops is a logistical challenge.

Of course, accountability matters. Longtime readers know my position on the tenure system, so I won’t beat that horse again, but it’s an uphill battle to sell disruptive change when people have the option of saying ‘no’ without consequence. The enemy isn’t really direct opposition; it’s foot-dragging.

I think that this line of thinking may also be part of why there is such a huge push for reform of teacher job security. Administrators are under enormous pressure to reform the education system and teachers may be very resistant.

Of course, one element that may be left out is that the teachers may be resistant to change for good reasons. When you have been in an organization long enough, you realize that a lot of reform can be about trying "old ideas all over again". These reforms can be both time consuming and ineffective. They may even lower outcomes due to the friction of implementation.

Let us consider a business analogy. One way that corporations tried to handle bad outcomes is a series of "re-orgs". These changes in structure have two good properties. One, the people in charge seems to be doing something to address issues by making changes. Two, a series of re-organizations can make it very hard to track a long term pattern of bad management as units break apart too often for performance to be easily tracked.

The ability of tenured people to resist cosmetic reforms is, obviously very frustrating to administrators who have little ability to influence the organization but seemingly unlimited accountability. However, endless re-organizations did not, in the end, help corporations like General Motors. Instead, they may well have accelerated the decline by focusing on making changes that were more cosmetic than effective. So do we really need to import the worst practices of modern corporations into the educational system?

[This post is also relevant -- Mark]

Outsourced blogging

I've been meaning to post something on the Collegiate Learning Assessment exam (which seems to be the main data source for the new book Academically Adrift, by Richard Arum and Josipa Roksa). In the meantime, Dean Dad has a good post on the subject.

Funding Cuts

From Dead Dad:

Of course, that refers only to general college reserves. It’s also common for various programs to have reserves of their own, earmarked for specific purposes. The college foundation might have reserves dedicated for certain scholarship awards. Some grant-funded programs will have reserves for specific functions and only for those functions. (In the context of multiyear grants, for example, it’s common to have ‘carryover’ of excess funds from one fiscal year to the next. That’s frequently allowed, but that doesn’t give license to transfer the extra grant money to the general college budget.) In cases like those, money comes with strings attached, and violating the terms of the money involves forfeiting the money. You can’t just move it around.

This lack of flexibility actually highlights one of the difficult issues with cuts in grants. If grants are cut by 10%, you don't have the discretation to eliminate 10% of the ongoing projects to make sure that the rest are successful. In the same sense, going after indirects to try and make up these losses would lead to the defunding of other operations.

The modern university budget looks remarkably inflexible to me, which makes planning for adverse financial circumstances appear to be extremely tricky. Not only do you have to cut but what you can cut can be remarkably constrained. This can lead to very poor optics (where something that looks non-essential is fully funded whereas a core operation simply lacks funding).

It is a tough place to be in!

Higher education and resource allocation

It is a powerful argument to reduce the concerns of your opposition by reframing their views into something that seems absurd.

Dean Dad writes:

Administrators may be the bearers of bad news, and sometimes the people who have to choose among terrible options. But to assume that we’re sitting on piles of money, cackling with glee while exploiting adjuncts and pocketing the savings for ourselves, is just otherworldly. It assumes a context completely out of keeping with anything I can recognize as reality. It’s so far afield that the only truly fitting rebuttal is a sigh.

I think that the issue is not that anybody is sitting on piles of cash. I think that there are two legitimate concerns. One, is the redistribution of resources (which is always going to be a feature of any organization that is not experiencing enough growth for all parties to "win" simultaneously). Two, is the concern about the growth of adminsitrative costs. Some of this growth is beyond the control of the administration (unfunded federal reporting mandates come to mind). But I think that it is a fair position to want to open a dialogue on this issue.

To dismiss these concerns out of hand doesn't seem to be the best way to encourage an open dialogue.