A huge conflict of interest -- more from Meredith Broussard's Atlantic piece on textbook companies and standardized tests

[The scheduler sometimes does weird things. This was meant to be the second of two excerpts. Sorry about the confusion.]

You need to read this:

Put simply, any teacher who wants his or her students to pass the tests has to give out books from the Big Three publishers. If you look at a textbook from one of these companies and look at the standardized tests written by the same company, even a third grader can see that many of the questions on the test are similar to the questions in the book. In fact, Pearson came under fire last year for using a passage on a standardized test that was taken verbatim from a Pearson textbook.

The issue often has as much to do with wording as it does with facts or figures. Consider this question from the 2009 PSSA, which asked third-grade students to write down an even number with three digits and then explain how they arrived at their answers. Here’s an example of a correct answer, taken from a testing supplement put out by the Pennsylvania Department of Education:

Here’s an example of a partially correct answer that earned the student just one point instead of two:

This second answer is correct, but the third-grade student lacked the specific conceptual underpinnings to explain why it was correct. TheEveryday Math curriculum happens to cover this rationale in detail, and the third-grade study guide instructs teachers to drill students on it: “What is one of the rules for odd and even factors and their products? How do you know that this rule is true?” A third-grader without a textbook can learn the difference between even and odd numbers, but she will find it hard to guess how the test-maker wants to see that difference explained.

Failures in targeted marketing

One of these days, I need to post a nice long thread on targeted marketing, benefits vs. limitations, promise vs. hype. In the meantime, I'll collect as many examples as I can of targeting done well, targeting done badly and, in this case, of targeting not done at all.

In a comment to Ken Levine's previously mentioned traffic report prank.

In Canada, Closed Captioning is sponsored and announced during the broadcast. Frequently, these sponsors are new album releases - Katy Perry, Rihanna, Coldplay.

Assuming that the language of the captioning is the same as the language of the broadcast and is not used for translation (not an absolute given, I suppose, in Canada), think about the target demographic of captioning.

Measurement

This is Joseph.

Thoreau:

We’re all familiar with the critiques of standardized tests and other common measures used for high-stakes decisions.  Recently, somebody in my circle has started going on about measures of “grit” and their predictive power.  I am willing to believe that “grit” is an excellent predictor of all sorts of things.  But I wonder if much of the predictive power of “grit” comes from the fact that these measures are currently low-stakes, so people have few incentives to game them.

 I really think that this is the heart of the measurement problem.  Insofar as there is a way to do better on a test, in a way that is less work than just be really good at it, then it is probable that much of your signal will be gaming.  Studying the form of the question, for example, is likely to improve performance (by less confusion, if nothing else) but access to these approaches may vary by context.

Even worse, some of the test prep may have nothing to do with the underlying measure.  So the score starts to measure things like "willingness to sacrifice learning time for test prep time". 

This is a very good insight and likely to be eternally problematic in education. 

Old story, new setting

I realize we've hammered this point quite a bit but, for their own good as well as the public's, charter schools have got to come up with an aggressive plan to deal with the self-dealing, price gouging and general looting that is becoming widespread in places like Michigan, Florida and Ohio.



From the Toledo Blade:

The charter school Imagine School for the Arts is paying rent of nearly $1 million a year on a downtown building with the education funding it gets from the state, prompting criticism from a progressive advocacy group that studied charter-school finances around the state.

The complicated financial arrangement also involves a school-affiliated trust company spending more than $7 million last year to buy a building valued at less than $2 million.

The liberal advocacy group ProgressOhio attacked the size of the rent payments at charter schools operated in Toledo and other Ohio cities by Imagine Schools Monday as excessive. Imagine is a national for-profit educational management company.

According to ProgressOhio, Imagine’s subsidiary, Schoolhouse Finance, collected at least $14.4 million in public money last year for the company’s 17 Ohio schools. Of that, $8.9 million covered rent for long-term leases to Schoolhouse Finance. The $5.5 million balance went to pay “indirect costs” to Imagine to provide management services.

Mark says smart stuff

This is Joseph

In the context of a recent post by Mark Palko:

Both the tech and financial sector have embraced the idea that economic rewards are directly correlated to work and worth. It's a strange mixture of efficient market theorem and social Darwinism, often with more than a bit of Randianism.

I found both of these arguments for increased taxation of high income earners interesting.   First:

Because rich people spend their money on useless stuff. Not far from where I live, there is a new house going up. It will be over 10,000 square feet when it is complete. 2,500 of those square feet will be a closet that has two separate floors, one for regular clothes and one for formal wear. If that is what you are spending your money on, then yes, I believe raising your taxes to fund education, infrastructure, and health spending is a net gain for society.

Don’t poor people spend money on stupid stuff? Of course they do. Isn’t the government an inefficient provider of some of these goods, like education? Maybe. But even if both those things are true, public investment and/or transfers to poor people will result in some net investment that I’m not currently getting from the mega-closet family. I’m happy to talk about alternative institutional settings that would ensure a greater proportion of the funds get spent on actual investments.

And,

Because I’m not afraid that some embattled, industrious core of “makers” will decide to “go Galt” and drop out of society, leaving the rest of us poor schleps to fend for ourselves.  Oh, however will we figure out how to feed ourselves without hedge fund managers around to guide us?

I think that this points out that the notion that people deserve the actual income/wealth that they currently have, in some sort of fundamental way, is a true measure of worth/contribution.  It is true that rich people do pay taxes, but consider all of the benefits they get?  We have a whole society of laws devoted to reducing kidnapping, murder of near kin, and robbery, and the wealthy definitely benefit from this. 

So I am not saying that the current level of taxes is too low, too high, or just right (on any specific segment of the market).  I am saying that an open discussion should consider issues like "everybody spends money to satisfy desires and that these desires rarely pass the scrutiny of outside parties".  Or that the whole idea of "going Galt" is often rather silly.  In academics, we have many good people for every position.  The marginal loss of any one person is sad but does not destroy the whole enterprise. 

Similarly, I suspect that there are a number of possible hedge fund managers in the world who could do a relatively comparable job (at least insofar as society as a whole matters -- losing your superstar investor could be a private tragedy).  Otherwise we'd expect the market to collapse when one of these key people dies of old age (and the market has been pretty robust to replacements via death so far). 

So some points to ponder. 

This is the perfect October clip

At least for a snarky epidemiology blog...

Graphs that make me nervous -- UPDATED

[Here's a good link to the full report.]


A couple of quick caveats: I am never entirely comfortable focusing too much on "additional years of learning" -- it's a problematic metric -- and the link to the original report seems to be dead.

The good folks at Vox are very excited by the results of this recent experiment in merit pay. Perhaps I'm missing something obvious, but that jump between 2010 and 2009 seems really big.

Does this look odd to anyone else?

Yes, a Surgeon General would come in handy right about now

For me, one of the most interesting stories in politics these days is the way that information has come to flow in the the conservative movement. And sometimes the most interesting part of that story is the way information fails to flow.

Rep. Jason Chaffetz (R-Utah) says Ron Klain is "off to a bad start" in his new role as the president's Ebola response coordinator, and that the U.S. Surgeon General should be the one leading the effort. But what Chaffetz doesn't seem to realize is that there hasn't been a surgeon general for more than a year.

“Why not have the surgeon general head this up?" Chaffetz asked in a Wednesday appearance on Fox News. "I think that’s a very legitimate question. At least you have somebody who has a medical background whose been confirmed by the United States Senate.”

“It begs the question, what does the surgeon general do?" he added. "Why aren’t we empowering that person?”

After this broke, Chaffetz tried to moonwalk his way back from the statement but there is simply no way to frame this so that the man comes off as both well-informed and honest. His problem is that he is trying to follow an official party line that makes consistency almost impossible (you can't block relevant nominations and gut relevant funding while plausibly complaining about the government doing too little to address an epidemic).

I suspect the root of the problem is that the leadership of the conservative movement fell in love with the appealing but doubly flawed idea that you can create optimal messaging by controlling the process. Fox News has always been a hothouse for ideas and arguments crafted to appeal to the base. Conflicting data and effective counter-arguments were largely kept out of the environment.

This approach can work for a while but at some point you lose control. The system is too complex to fine-tune. Eventually you find yourself saddled with a bunch of ideas that the base is committed to even though they can't hope to survive in the outside world.

"—We Also Walk Dogs"

I've been going through some old posts on incentive pay and they got me thinking about how education reformers, fresh water economists and libertarians often fall back on a strongly linear model. That, in turn, got me thinking about a Heinlein story I read years ago.

From Wikipedia:

"—We Also Walk Dogs" is a science fiction short story by Robert A. Heinlein. One of his Future History stories, it was first published in Astounding Science Fiction (July 1941 as by Anson MacDonald)....

'General Services', a very successful company that provides various personal services ... is asked to ... enable an interplanetary conference to be held on Earth, whose strong gravity is inhospitable to many of the native races of other planets in the solar system.

Much of the action of the story is ... about how to persuade the world’s leading physicist to undertake the job.

As memory serves, that really is the gist of the story. Find the right person, offer him or her sufficient incentive, get what you want. Add to that an entrepreneur hero...

Optimization and college enrollments

This is Joseph.

Dean Dad had a great post a while back on trying to improve classroom scheduling.  In it, he had a great example of the perils of overfitting.  The context is that the Gates Foundation had suggested that a more efficient algorithm for classroom management could improve costs.  The problem is that they aren't testing this in prospective environments -- leading to some issues:

Finally, and I can’t stress this enough, enrollment is a moving target.  Patterns change unevenly, so in any given year you’ll get a fresh new crop of anomalies.  When enrollment continues until just before classes start, as is typical at community colleges, you’re left making predictions based on partial information.  And you have to make those predictions early enough for alternate plans to be realistic.  That’s never a perfect science -- every year, someone complains that his course would have filled if we had just given it more time, which is unprovable either way -- but it’s inherent in being “responsive.”  If we locked down enrollments months in advance, we would have time to squeeze out some efficiencies.  With numbers changing until the last minute, it’s harder.  You’ll never capture that if you only look at “snapshots.”

It's also worth noting that the staffing side is also an issue.  Paying for "Just in time" staffing for a community college would almost certainly reduce the qualifications of the instructors, do strange things to the workload of permanent staff, or increase prices.  After all, if you do not know if you are going to be taking a class until the first day of classes (when enrollments are likely close enough to know) then you will likely charge more or look for another job.  Landlords are notoriously unsympathetic to "most of the time this works and I can pay rent" pleas.

So the report might be interesting as an estimate of the maximum possible benefit.  But in the real world you never get 100% efficiency in any process, especially one that has ebbs and flows.  Even McDonalds, good as they are (and they are really good at this) cannot completely eliminate wait times due to the lunch rush. 

Mark Evanier points out the fundamental contradiction in conservative complaints about Hollywood's "political correctness"

Mark Evanier is one of LA's leading pop culture experts (and in this town that's a high bar). He's been writing for television and comics for around forty years now and probably writing about them for even longer so when it comes to things like the process of creating cartoons for television, he speaks from a position of authority.

Something that can't really be said for Tucker Carlson,

Nothing is scarier to a modern liberal than tobacco. If Popeye were driving around giving the morning after pill to fourth graders that would be totally fine. But smoking a pipe, a symbol of freedom and masculinity in America itself, the reason this country exists, tobacco, that's like, "Oh, that's outrageous. That's a major sin."

As you probably guessed, Carlson was ranting about a clip from a proposed Popeye cartoon (from the gifted Genndy Tartakovsky). He was also misrepresenting the way cartoons get made. Evanier sets him straight.

The other thing Mr. Carlson doesn't get about it is that this is not a political decision. It's a marketing concern. Most of the time when someone decides to launder animation — tone down sex or violence — it's because they want to make sure they can sell reruns of the product to the widest audience for the longest time.

About ten years ago, I was approached about writing a Popeye animated project. It never happened but we had a meeting or two and it was made pretty clear to me that merchandising and marketing were driving this particular venture; that any decision about Popeye having a pipe or Popeye punching out Bluto would be decided on that basis. They were going to make some assessment based on an estimate of how many parents wouldn't buy a toy if the character had a pipe…or how many countries or networks wouldn't buy and air the show.

A principle Tucker Carlson has voiced as long as I've followed him on TV is that companies should be free from regulation and outside pressures that might minimize their profit margins. And now here he is, arguing that the people who are marketing Popeye need to give Popeye his pipe. Someone needs to ask Carlson, "Even if they think they'll make less money if they do?" Because if they don't give Popeye a pipe, that'll be the reason.

... That's all that's involved here. No politics. No hidden agenda. Just someone's idea of how to exploit a property.

There has always been a significant enemy-of-my-enemy quality to the alliance of social and pro-business conservatives. The small town pastor and the big city banker never really trusted each other but their interests often aligned.

What has changed is the nature and scope of right-wing media. A list of the causes would probably be the stuff of a Ph.D. thesis, but it would certainly include the explosion of cable channels and the death of the Fairness Doctrine. Whatever the reason, there is now a major industry built around supplying hundreds of hours a week of a very specific kind of news and commentary.

Fox News is targeted mass media. What's more, it's targeting people who feel abandoned, even disenfranchised by the media at large (yes, I realize we're talking about sixty-year-old, upper and upper-middle class white guys, but we're also talking about perceptions). Rants about Popeye's pipe play well with that audience, as do most complaints about Hollywood liberals.

The argument of media liberal bias has always had a problem reconciling the position with the cost factor ("Freedom of the press is guaranteed only to those who own one." -- A.J. Liebling), but fifty or one hundred years ago when moguls acted more independently, there were examples of rich liberal moguls who were willing to sometimes put principle and even whim above profits (Turn of the Century Hearst comes to mind). In the Twenty-first Century, the vast majority of media is controlled by a handful of huge corporations. The suggestion that these companies are putting a progressive agenda ahead of profits strains credulity.

Three films

Sometimes, in the course of researching a post, a link leads to a completely unexpected corner of the internet. In this case, a post on Mark Evanier lead here.

The first is a cameraless film with hand-drawn sound from before the war, the second explains the technique and the third stands alone.


Norman McLaren - Dots (1940)

Have you had your flu shot yet?

This is Joseph

I was reading Mike the Mad Biologist's web page and I noted this article on Ebola and the flu.  Ebola has been in the news a lot but influenza remains a bigger killer than Ebola:

Ebola has claimed fewer than 4,000 lives globally to date, none in the United States. Flu claims between 250,000 and 500,000 lives every year, including over 20,000 in the United States—far more American lives than Ebola will ever claim.

Notice just how terrible the Spanish flu was:

It infected 500 million people across the world, including remote Pacific islands and the Arctic, and killed 50 to 100 million of them—three to five percent of the world's population

Today, that would be a disease that killed between nine and fifteen million Americans.  And the easiest option to reduce risk is to vaccinate against the infection. 

Have you had your flu vaccine?  [I get mine annually, usually on the first day it is permitted]

When journalists have an ethical obligation to roll their eyes...

...or at least ask a follow-up question.

Maybe it's just me, but don't journalists have an obligation to object when sources say something blatantly, laughably untrue? If the subject of an interview has just insulted the intelligence of your readers, should you at least attempt to get a more truthful response?

Andrew Cuomo has a new book out. To promote it he has a brief interview with Amy Chozick in the New York Times. I realize that this isn't a setting where you expect hard-hitting journalism, but this still stands out.

You also write that there are ultimately more negatives than positives to being a famous politician’s son. Why? I wouldn’t trade my father or our experiences for anything. But politically, it’s a negative, because you get all your father’s enemies — and not all his friends.

As Scott Lemieux observes, "It’s amazing how many public officials in this land have managed to get beyond this handicap anyway." Everyone knows that Cuomo's background gave him an enormous advantage. He knows it. Chozick knows it. Her editor knows it. We know it.

However, Cuomo also knows that no no one at the NYT is going to rock this particular boat. It's possible that Chozick asked some kind of follow-up -- the article includes the rather troubling disclaimer "INTERVIEW HAS BEEN CONDENSED AND EDITED".-- but the piece that I saw just moves on.

David Brooks, con(firmation) artist

[picking up where we left off here]

David Brooks has one of the most desirable jobs in journalism; it is also one of the most difficult. He has to present a mainstream conservative perspective in a way that appeals to his center-left target audience (which includes but is somewhat larger than the target audience of the New York Times). His job is greatly complicated by shifts in the Republican Party and the conservative movement. On a growing number of issues there is less and less common ground to focus on.

Fortunately for Brooks, there is a (hopefully unwritten) loophole in the professional standards of the NYT and many other publications which allows writers to break all sorts of rules -- up to and including the one against making things up -- as long as these lapses serve to reinforce the readers' (or at least the editors' and publishers') preconceptions.

The target audience for the NYT is mainly in the top half (possibly top quartile) in terms of wealth and education. They may not be Randian but they still tend to look at the bottom quartile with an attitude that is by turns suspicious and patronizing. Check out how Brooks plays to those prejudices in this recent column (via Charles Pierce).

In the first place, we’re living in a segmented society. Over the past few decades we’ve seen a pervasive increase in the gaps between different social classes. People are much less likely to marry across social class, or to join a club and befriend people across social class.

That means there are many more people who feel completely alienated from the leadership class of this country, whether it’s the political, cultural or scientific leadership. They don’t know people in authority. They perceive a vast status gap between themselves and people in authority. They may harbor feelings of intellectual inferiority toward people in authority. It becomes easy to wave away the whole lot of them, and that distrust isolates them further. “What loneliness is more lonely than distrust,” George Eliot writes in “Middlemarch.”

So you get the rise of the anti-vaccine parents, who simply distrust the cloud of experts telling them that vaccines are safe for their children. You get the rise of the anti-science folks, who distrust the realm of far-off studies and prefer anecdotes from friends to data about populations. You get more and more people who simply do not believe what the establishment is telling them about the Ebola virus, especially since the establishment doesn’t seem particularly competent anyway.

This is brilliantly Brooksian. The first paragraph is accurate and well-grounded. The second is where things start getting subtle. The transition from fact to inference is so smooth and the conclusions he draws are so reasonable that few readers will realize how skillfully they've been set up.

Note the patronizing, more-to-be-pitied-than-censured tone. In his detached, scholarly, Middlemarch-quoting way, Brooks sympathizes with those in the lower class. He understands how conditions have made them angry, resentful and suspicious and how those emotions have made them gullible and prone to hysteria.

Having framed the question in such reasonable terms and having masterfully played on the prejudices of his readers, Brooks feels free to start making things up. Pretty much all the statements in the third paragraph are lies of varying degree, but Brooks knows they are lies most of his readers were predisposed to believe.

Working backwards (which allows us to save the best for last), let's start Ebola. Brooks is basically talking about the sort of thing we've been hearing from this guy (to get the full effect, go back and read the second paragraph).

How about the broader anti-science question? Brooks credits it to the great unwashed sharing misinformation but he offers no data to back it up, but if we look at the most obvious example (denial  of global warming), there is an alternative explanation that does have considerable evidence to support it.  George Will also figures prominently in this one.

And finally, how about the resentful and suspicious lower classes driving the anti-vaccination movement?

Not so much:

Exemption rates vary greatly by area and school. Los Angeles Unified kindergartens, for example, had an overall exemption rate of just 1.6%, although there are several in the district where more than 8% of students have belief exemptions. At Santa Monica-Malibu Unified, the overall exemption rate was 14.8% and at Capistrano Unified in south Orange County, it was 9.5%. At nearby Santa Ana Unified only 0.2% of kindergartners had exemptions on file.

In Los Angeles County, the rise in personal belief exemptions is most prominent in wealthy coastal and mountain communities, The Times analysis shows. The more than 150 schools with exemption rates of 8% or higher for at least one vaccine were located in census tracts where the incomes averaged $94,500 — nearly 60% higher than the county median.

Brooks has been doing this sort of thing for a long time now, long enough for the NYT to know what's going on..