The two big assumptions behind 'best practices'

[Homonym alert -- another smart phone composition]

There is no notion better loved by scientific management types than 'best practices.' It has become one of the truly obligatory PowerPoint phrases and it is rare to hear a high-level executive lay out plans for fixing a large organization without promising to promote them. These days those executives often have titles like 'superintendent.' Scientific management is now extraordinarily popular in education circles, particularly in the education reform movement. This confluence of two of the major interests of this blog means there are some major threads starting that will allude to this topic quite a bit.

'Best practices' are often treated as a self-evidently good thing  – Who wouldn't want to be best? – – but the idea of finding optimal methods and strategies and applying them across an organization is based on two very big and difficult-to-justify assumptions.

The first is that you have or even can identify these behaviors. Determining "best" status usually requires a high degree of faith in your metrics and metrics almost never perfectly align with the properties they are supposed to measure. Even when handled by the most competent of people, things can go badly wrong.

And, as anyone who has worked for large corporations can tell you, the executives overseeing these programs often are not up to the job. Between various biases, office politics, and a widespread misunderstanding of how statistics works, the results are often far from "best."

Most corporate cultures strongly discourage employees from questioning whether or not a "best practice" is actually best, but at least the question does sometimes come up. The very idea of the next assumption often escapes people entirely.

In order for a best practices approach to make any sense whatsoever, the optimal level of the factors in question must remain basically the same from person to person, location to location, and sometimes even job to job. Those are extremely strong and in some cases wildly counterintuitive assumptions and yet they go unquestioned all the time.

Later in the week, we'll discuss how violating these assumptions can lead to trouble (and connect this topic with an earlier post).

Freakonomics

This piece of a review of the newest Freakonomics book is a bit harsh:

They take this as evidence that even intelligent politicians don't like hearing uncomfortable truths that challenge positions to which they are committed. But it seems more likely that Cameron, who is indeed an intelligent politician, noticed they were talking nonsense. After all, it's a ridiculous analogy. People don't go to the NHS and "pick out" their treatment. They are in the hands of doctors and other healthcare professionals who collectively try to find the best treatments for them, within limits. Healthcare is nothing like transportation. If it were, the NHS, whatever future problems it might be facing, could hardly have survived so long (and performed more efficiently than the rival US system, where many patients really are "picking out" their preferred treatments). Only two economists (or rather one-and-a-half economists) could be so arrogant and so ignorant as to think that this was how to talk to a future British prime minister about healthcare. I imagine that what Cameron was really thinking was: if these are the clever people, spare me from the stupid ones.

But it does get at a very real issue -- different markets operate in different ways.   Noah Smith and Paul Krugman point out the difficult link between theory and empiricism here.  It is true that very few health care markets are purely socialist or free -- they are classical mixed markets in most places.  That said, the English model is hardly a disaster. 

What I generally want to see when people suggest large countries radically change an existing system (upsetting many stakeholders and people who planned their lives around a set of rules) is that there will be relatively immediate benefits (i.e. it is politically viable) or long term improvements for the country (i.e. it is worth losing an election over as it makes people's lives better). 

Levitt clarified his ideas, but I think they still have some serious issues.  He suggests:

On January 1 of each year, the British government would mail a check for 1,000 pounds to every British resident. They can do whatever they want with that money, but if they are being prudent, they might want to set it aside to cover out-of-pocket health care costs. In my system, individuals are now required to pay out-of-pocket for 100 percent of their health care costs up to 2,000 pounds, and 50 percent of the costs between 2,000 pounds and 8,000 pounds. The government pays for all expenses over 8,000 pounds in a year.

Yet, ironically, this approach requires a real faith in effective government.  Why?  Because there will be people who need different amounts of subsidy (currently poor elderly, for example, or those whose illness prevents them from working).  Or it requires getting rid of universality, which might be a feature in the long run but has many bad features in the short run.  It adds in layers of billing and pricing, that health systems are often poor at generating (or at least the US systems seem to be).  People need to be able to get accurate price quotes, collections for debt needs to exist, and the hospital have to set up payment under difficult circumstances. 

It also requires the government to dynamically adjust the "payout" and the "thresholds" as prices change.  Will that be done by a central payments board?  How is that really different than the current English approach of determining cost effectiveness in general?    You are still using expert opinion to run the system, just in a different place.  And there could be some real concern that the "thousand pound" subsidy could be used to replace current benefits or drop over time as it is eroded by inflation. 

So what is the evidence that this radical reform will reduce total health care costs?  Because this is a case where we want to have a clear test of the theory before completely reforming a relatively functional system (it's mediocre for the OECD but one can go down from average, as well as up).  There are some examples of health care systems that might be more free market plus regulation oriented (say Singapore).  But why not the French model? 

I'd like for you all to take a look at something...

This is one of those big, complex stories that are themselves subplots of still bigger, more complex stories. These subjects tend to get out off if you approach them too ambitiously. The best way to handle them, at least for me, is to keep each post small and manageable, at least while you're laying the groundwork (blogs are good for that).

I've been meaning for a long time to write about Relay Graduate School of Education and its role in the education reform movement. Lots of moving parts here, but there's one very important part that I think we can pull out and address as a stand alone question:

Does getting a master's from Relay require graduate level work?

When you have a minute, check out Relay's website then watch a few of these videos. They're all short and relatively self-explanatory. I don't want to go into content, pedagogy, or applicability at this point (that can wait until later posts), but I do want to mention that the most interesting (hooking the lesson) is also the least representative.

Here are some of the videos which, as far as I can tell, are used as part the school's flipped classes:

Lesson plans*


Transitions


Wait Time


Clear and precise directions


Strong voice and positive framing


Hooking lessons


It's been a long time since I took a methods class but this looks a lot like what I remember from sophomore and junior level courses. Are these things you would normally see at a graduate level? I realize these videos are only part of the program but this description of the curriculum does nothing to allay my concerns:

The Relay GSE curriculum comprises two core components. First, graduate students learn core instructional practices in planning, delivery, and assessment of teaching and learning that are necessary for all teachers, regardless of the subject or grade level they are teaching. These practices are sometimes referred to in the education field as “general pedagogy.” Second, graduate students will learn how to teach their specific subject at one or more specific grade levels. Math teachers do not learn geometry, for example, but rather how to teach geometry and how students learn geometry. This is sometimes referred to as “pedagogical content knowledge.

Help me out on this one. Based strictly on the information Relay provides us, would you say this constitutes graduate or undergraduate level work?


*Not sure if intentionality (intensionality?) means what he thinks it means.

Quoting Pólya quoting Shaw

Along with Richard Feynman's "total temperature of the stars," this is my favorite comment on 'new math.'


From The Random Walks of George Pólya:

In the immediate post-Sputnik era Pólya had been an outspoken critic of the formalism of the School Mathematics Study Group (SMSG) and the “new math." He often cited an example taken from the SMSG geometry text that gave a theorem with proof—taking up half a page—stating that with three points on a line, one point must lie between the other two. He argued that though this is a necessary theorem for a foundations course in geometry, it has no place in an elementary text. He said that had he been asked to study the proof of such a theorem in high school, he would almost certainly have given up on mathematics, having concludcd that the subject is dumb! In support of this viewpoint— that there was excessive rigor in the “new math"—he was one of the signers of a manifesto on curriculum reform that appeared in The Mathematics Teacher and The American Mathematical Monthly decrying the direction of the reform movement of the 50's and 60's.

In talking about what he regarded as the excess of rigor in SMSG. he cited the oft‘repeated story about Isadora Duncan's proposal of something like marriage to George Bernard Shaw. She argued that their children would have his intelligence and her beauty. Of course, Shaw pointed out that the children might well have... Pólya suggested that SMSG had been put together by research mathematicians and high school teachers. on the assumption that the material would reflect the mathematical sophistication of the researchers. and the pedagogical skills of the high school teachers. But then, like Shaw, he pointed out that the material in fact reflected... The observation was unkind but there was, perhaps. some truth in it.

Weekend blogging -- Breaker Morant

I've always meant to see this one, even more so since I came across Edward Woodward's extraordinary work as the self-loathing agent Callan.

"Santa Ana winds blowin' hot from the north..."

Yes, it's been one of those weeks.

“There was a desert wind blowing that night. It was one of those hot dry Santa Anas that come down through the mountain passes and curl your hair and make your nerves jump and your skin itch. On nights like that every booze party ends in a fight. Meek little wives feel the edge of the carving knife and study their husbands' necks. Anything can happen. You can even get a full glass of beer at a cocktail lounge.”

― Raymond Chandler, Red Wind: A Collection of Short Stories

The firefighters say they've never seen them coming in like this.

The SAT probably is unfair to the disadvantaged but not for the reasons you've been hearing

This is another one of those posts that I started weeks ago as part of the big SAT thread then didn't get around to posting. One of the big questions was the fairness of the test. I had alluded to the problem but usually as a side question. Part of the reason I didn't spend more time on the question was because of my specialty. Though I did originally certify to teach math and English, I've been focused almost entirely on the former for a number of years, and almost all of the prep work I've done with student has been on the math side.

If I had been working with the verbal part, I would have had to address some uncomfortable questions. The verbal SAT is a good, well-designed and informative test but there are inescapable concerns about its fairness.It is very difficult to design a verbal reasoning test that is not culturally biased. Language and culture are so intertwined that it is almost impossible to even discuss one with out considering the other. Cultural biases are not nearly as much of a concern on the mathematics side of the test.

Still, even in the reasonably objective and unambiguous world of mathematics, there are any number of ways in which background can give an unfair advantage. These include (but are by no means limited to) enrichment activities, role models, high-achieving schools and community culture, support and tutoring, and expensive prep classes.

This last item has become one of if not the central element in the SAT/fairness discussion. All stories on the subject seem to be contractually obligated to talk about expensive test prep courses and yet, as far as I can tell, they all frame the issue in a way that makes the criticisms completely invalid.

There are two fallacies in this standard line of argument: The first is based on confusion over absolute versus relative values; the second is based on a common but profoundly wrong concept of the test itself.

As an absolute statement it is true that if prep courses do any good at all then the ability to pay for them will provide an unfair advantage. The question on the table now, though, is not absolute. The people who are arguing for the elimination of the SAT are also arguing, sometimes implicitly, often explicitly, for grades to take a much larger role in the college selection process to take up the slack. This leads to a very different question: does having money give one a greater advantage on the SAT then it does on GPA?

Private tutoring centers are a huge national industry, and if you send your child to one for any length of time, the cost will probably be far greater than what you would've spent on an SAT prep course. We could, of course, have a long discussion about the intrinsic value of what is taught in one versus the other, but from an economic fairness standpoint, all we care about is the cost and the effectiveness in improving the given metric.

A valid argument here would start with a comparison of the ways that privilege can provide an unfair advantage on the SAT versus GPA, but what we've gotten so far is the pseudo-argument: A is worse than B because A is bad ("French Fries are so fatty; I think I'll have onion rings instead."). As far as I can tell, none of the many stories describing the potential impact of prep courses even mention the existence of the private tutoring industry.

The other fallacy here is the very wrong but very common belief that the SAT is some kind of mysterious black box, the secrets of which can only be revealed by one of the illuminated. I've already been through this at some length but just to reiterate, because of the stability of the tests and the large number of previous editions that the College Board has published, the SAT is one of the most transparent exams you're ever likely to take.

At least on the math side (which is the area I have some experience with), this transparency, along with the nature of the questions, makes the test surprisingly easy to teach and to teach yourself. In the latter case, it goes like this:

Take one of the old tests (don't worry about the time limit);

Check your answers;

Read the explanations for the ones you got wrong;

If you don't understand the explanations for some of those problems, take them to a teacher or administrator and ask for help (as a former teacher, I can tell you that educators love to see this kind of initiative and will go to great lengths to encourage it). There are also free after-school programs that would be glad to help (I volunteer at one of them);

Repeat the process. After you start breaking fifty or sixty percent, work on reducing your time.

If it's this easy, why does anyone bother with a pricey prep course? Well, for one thing, it's not that easy. We are talking about a tremendous amount of work and self-discipline. The courses provide structure and external discipline, not to mention a large dose of motivation and reassurance to counteract the test's foreboding reputation (a problem greatly compounded by journalists' tendency to talk about the exam in dark and mysterious terms).

To sum up, there is tremendous unfairness in our education system. The SAT is sometimes part of that unfairness, but neither for the reasons or to the extent you often hear.

A cyncial take

Kevin Drum asks:

Anyway, that's my question. There's already a perfectly good, perfectly simple way for ISPs to recover the cost of providing lots of bandwidth: just charge the customers who use it. Existing peering and transit arrangements wouldn't be affected, and there would be no net neutrality implications. So why not do it? What am I missing?

My cynical answer is that there are a lot of markets that are large but for which service is sub-optimal (think New York City)  If you charge users by bandwidth, the people in these markets would likely end up getting a discount because they are light user simply because it is nearly impossible to be heavy users.  But everyone would like some internet access.

So this is a way to have your cake and eat it as well.  In markets with bad service you make money as a gatekeeper.  In places with good service, you recover cash from the content providers who use the capacity. 

If you are wondering if this sounds a lot like a monopoly or a lack of competitive markets, you are probably correct.   After all, cell phones (which appear to still have a competitive market) had absolutely no trouble rolling out bandwidth based pricing.  Customers grumbled, but everyone gave up and adapted to it. 

Another one for the West Coast Stat Views lexicon: Jethro Models

[Part of an ongoing series]

The Jethro Model is a formal or informal model that leaves out a large number of necessary parts. The allusion was explained in a previous post.

From Anti-orthogonality at Freakonomics

In one of the many recurring gags on the Beverly Hillbillies, whenever Jethro finished fixing the old flatbed truck, Jed would notice a small pile of engine parts on the ground next to the truck and Jethro would nonchalantly explain that those were the parts that were left over. I always liked that gag and the part that really sold it was the fact that the character saw this as a natural part of auto repair: when you took an engine apart then reassembled it you would always have parts left over.

Sometimes I find myself having a Jed moment when I read certain pop econ pieces.

"What's that pile next to your argument?"

"Oh, that's just some non-linear relationships, interactions, data quality issues and metrics that won't reduce to a scalar. We always have a bunch of stuff like that left over when we put together an argument."

For a recent example, consider this quote from  George Mason University economist Robin Hanson (via Andrew Gelman):

If your main reason for talking is to socialize, you’ll want to talk about whatever everyone else is talking about. Like say the missing Malaysia Airlines plane. But if instead your purpose is to gain and spread useful insight, so that we can all understand more about things that matter, you’ll want to look for relatively neglected topics. . . .

Obviously, this is intended more as an observation than even an informal model, but we're still looking at a level of simplification that makes this rule pretty much meaningless; as soon as add any of the complexity of actual conversations, either with respect to why we converse or how we decide what to talk about, the whole argument just collapses. We converse for a long list of reasons. Sometimes we simply want company. Other times it's something more specific, to propagate our ideas, to amuse, to impress, to be liked, to establish individual and group identity, to get laid or, far more frequently, to convince ourselves that we could get laid if we wanted to. We could make similar list of reasons for picking conversational topics, but I think you get the point.

To reduce this down to social vs. informative motives and common vs. neglected topics, you either have to leave out important options or group together things so diverse as to make the definitions meaningless. What's more, by equating neglected topics with informative conversations, the model suggests some strange implications, such as that the person who just wants to be sociable will talk about racism and climate change, while the person who wants to be informative is more likely to discuss obscure distinctions between Phish bootlegs.

That's not to say that there's no extra value to bringing up neglected topics; it's just that Hanson's observation doesn't capture the fundamental relationships. I've been writing quite a bit recently on the importance of orthogonality and there's certainly a relationship between unique information and how much a topic has been discussed. Unfortunately there's also a great deal of collinearity. Lots of topics are relatively neglected because they don't contain that much interesting information.

To further complicate matters, under the right circumstances, you can gain considerable social cachet by knowing interesting facts about little known topics. The "interesting" part can be a bit of a hurdle, but I know  people who do it which puts yet another hole in the model. As do people who bring up obscure topics for the primarily social purpose of making themselves seem distinctive or erudite.

Another problem with Jethro models is the way that their oversimplified, overgeneralized approach can enable self-serving hero/villain narratives. Andrew Gelman made a related point about many popular economics books and articles -- "What strikes me about this discussion is the mix of descriptive and normative that seems so characteristic of pop-microeconomics." You don't have to look hard to see that mix here -- you can almost hear the inspirational music in the background while reading this "if instead your purpose is to gain and spread useful insight, so that we can all understand more about things that matter, you’ll want to look for relatively neglected topics."

It should be noted that Robin Hanson spends a great deal of time on out-of-the-mainstream ideas. Without putting too fine a point on it, when someone who "has elected to have his head cryonically preserved in the event of medical death" depicts in such glowing terms people who discuss neglected topics, I can't help but suspect bias.

And given Hanson's tendency to portray himself as being above this sort of thing...

Yes, the House is still capable of bipartisan action...

...but what's interesting is where they choose to do it. Charter schools certainly aren't the least controversial issue facing congress and, if anything, they've become more so as stories have poured in about waste, huge payouts, discrimination, draconian discipline policies, and community protests. I don't want to demonize charters here -- there are a lot of good ones out there and I think they have an important role to play -- but they don't seem to be the sort of issue that could manage a 360 to 45 vote.

The answer lies, I think, in two factors: first, that it's easy to get a charter school bill in under the radar; and second, that charters have wide support on both the left and the right where it counts, in the media and among wealthy donors.

From the Hill:

The House on Friday passed bipartisan legislation to expand access to charter school funding.

Passed 360-45, the vote came in sharp contrast to the bitterly partisan debates this week over creating a select committee to investigate the 2012 Benghazi attack and holding former Internal Revenue Service official Lois Lerner in contempt of Congress.

A majority of Democrats — 158 in favor and 34 against — joined all but 11 Republicans in support of the measure.

The bill authored by House Education and the Workforce Committee Chairman John Kline (R-Minn.) and the panel's top Democrat, Rep. George Miller (Calif.), would consolidate the two existing federal charter school programs into one to award grants to state entities.

The measure would also authorize the secretary of Education to maintain a federal grant competition for charter schools that did not win state grants.

Republicans have touted the issue of school choice and access to charter schools as a way of limiting the federal government's role in education policy. Charter schools receive public funding, but operate independently and therefore are not subject to federal regulations.

"Expanding education opportunity for all students everywhere is the civil rights issue of our time," House Majority Leader Eric Cantor (R-Va.) said. "I say we help those students by expanding those slots so they can get off the waiting lists and into the classrooms."

Weekend blogging -- brought to you by the Good Wife

One of the small but ubiquitous changes the internet has brought is the end of the lost song. Before the mid-Nineties, the main ways to learn the names of songs was from the DJs who sometimes remembered to tell you who you were listening to or from the captions on videos that had a way of fading just as you remembered to look up. Songs you heard on TV shows and movies were generally lost causes. The irritating feeling that came from not being able to find or forget a song (specifically "Anna" but not the Beatles cover) was the basis of at least one sitcom episode. From Wikipedia:

In the Married... with Children episode "Oldies But Young 'Uns" (Season 5, Episode 17; airdate March 17, 1991), Al Bundy becomes obsessed with finding out the name of this song which has become his earworm (originally he can only tell people the nondescript misheard lyric "hmm hmm him").

It is still possible not to be able to find a song, but it doesn't happen often. If you can remember a fragment of a lyric or pin down where you heard it, you can usually be listening to it on Youtube in a couple of minutes.

On last week's The Good Wife, a distinct and very catchy beat kept running through the episode. As soon as it was over I went online and learned that the beat came from the equally catchy song "High On the Ceiling."






Once I got on the subject, I remembered an obscure song from Malcolm in the Middle. Googling the show's title and the word 'hockey' was enough to bring up the song.




"Little Buster" from the beloved coming of age anime FLCL was another potential earworm that proved easy to find.



I have to admit, I could never get into that show. The only anime I ever really connected with was Cowboy Bebop but that one won me over completely. It also had one of the great late 60s/early 70s opening titles. Even Lalo Schifrin would have been jealous.



Technically, this last one doesn't exactly belong on this list -- I was already a big fan if the song -- but I like this version a lot and, like all good covers, it reveal something interesting that you probably missed in the original.

A musical introduction to the old "new math"

The commonality between the current education reform movement and the Post-Sputnik era have been mentioned before. Among other similarities, both movements prided themselves on taking a rigorously scientific approach to education and yet some of their sharpest critics were the very scientists and mathematicians they were trying to emulate.

We've already talked about Richard P. Feynman's criticism of Post-Sputnik era math and science textbooks, particularly their attempts to be rigorous and realistic. Along similar lines, Tom Lehrer, who was either teaching math at Harvard or political science at MIT (depending on exactly when the song came out), had a great deal of fun with the topic in the song "New Math."

Two more for the West Coast Stat Views lexicon: The Jar Jar Binks Paradox and Mathematical Anosognosia

The Jar Jar Binks Paradox

Improving the reputation of something bad by adding an additional element that's even worse. The effect works by focusing criticism on one point, making the other elements look better by comparison, and by creating a more favorable narrative (____ would have been good if not for ______).

You could argue that the fatalities-per-mile metric was the Jar Jar Binks of Freakonomics' shoddy analysis of the risks of walking drunk vs. driving drunk. Just to be clear, walking drunk is very dangerous. It might even be more dangerous that driving inebriated, but Levitt's analysis was a collection of comically oversimplified assumptions and numbers pulled out of the air. (See here, here and here for critiques). By addressing criticisms of the fatalities-per-mile metric, Levitt was able to create the impression that the rest of the work was solid.


Mathematical Anosognosia

A condition that causes the false impression of comprehension when a concept is accompanied with familiar mathematical symbols and methods. This is often accompanied with a heightened sense of self-confidence and diminished sense of judgement and restraint. Those prone to this condition are often observed making sweeping pronouncements in fields they have no relevant background in. Though almost anyone working in a math-based field can suffer from Mathematical Anosognosia, physicists and economists seem most susceptible, Extreme cases have been known to produce NYT best-sellers.

More futures past -- Highway edition

The standard post to accompany this sort of clip either oohs and aahs over the more prophetic aspects or laughs at the less realistic, but I don't feel like either of those fits my reaction. What strikes me watching this is how fast-moving and, more importantly, ambitious people expected the future to be. This was particularly notable in the section describing road construction (living in LA no doubt contributes to my reaction).

When I look at the Post-War era, I almost always get this incredible sense of pent-up energy, as if the country couldn't wait to make up for all those years lost to the Depression and the War. People wanted to do big things. What's more, they wanted to do them as soon as possible and they were willing to pay whatever was required.

It would be interesting to try to attach some numbers to the attitudes, but just anecdotally it seems clear that when it comes to progress, we're now more tolerant of delay and less tolerant of cost. When an ambitious proposal (manned space exploration, hypersonic trains) does make the news, it almost invariably comes with a laughably low-balled cost, usually one or more orders of magnitude below reasonable.

We'd still like magic highways, just not enough to foot the bill.




Disney's Magic Highway - 1958

A distributional question

I have been under the weather for a bit (thus no posts) but I wanted to share a thought I have been having in reaction to the minimum wage discussion on the west coast.  People have tended to be worried about these increases, with much concern about economic damage.

However, income and wealth in the United States are distributed with an extremely heavy tail, especially in terms of growth in the last 30 years.  This sort of growth, presuming a perfect market, is quite odd as I had always presumed human ability has a normal distribution.  The normal distribution is continuous and naturally presents several people of nearly the same ability behind the exceptional person (at least at the top end).  We can ignore the odd outlier -- if there was only one billionaire in the United States and they had had risen from poverty then we'd exclude them from any realistic analysis. 

But if we want to argue that current economic trends represent fairness, it turns out that we have some steep assumptions and observations to explain. These turn out to be crucial.

For example, when top wages are so high, why don't the companies hire the "next best executive" and split the surplus?  If ability is an innate (as opposed to learned ability), why don't we outsource all CEO jobs immediately to China (which would have more top performers as a function of a larger population)?  But this sort of wealth distribution seems like an odd way to end up given a normal distribution of ability, presuming one is talking about some sort of meritocratic environment.

Or how do we know we have an ideal marketplace now?  There have been a lot of commercial structures over history.  What makes us different than: a) Plato's Athens, b) the Roman Empire, c) Medieval England (say the anarchy period), or d) the Song dynasty in China?  They had markets too -- where the results of such markets just?  If the difference is due to corruption, government interference, and rent seeking, do we have a better balance now?  And how would we know, without making a consequentialist argument? 

It is actually a pretty deep question.