The word 'metacognitive' is usually a bad sign

Lucy Calkins of the Reading and Writing Project at Teachers College has an article on something I should probably be paying more attention to.

In April, hundreds of thousands of children in grades 3-8 in New York State took Pearson’s version of the totally new Common Core aligned literacy exams. The students’ scores on those tests have yet to come out, but the New York State Department and Pearson’s scores have been accumulating as well, and accounts of how well they’ve done on this test are not good. Each week, another journalist produces a blog or a column that begins, “I recently obtained a bootlegged copy of the New York State English Language Arts (ELA) exam…” and then the article proceeds to critique the exam. One recent journalist, for example, wrote, “I received full copies of each of the sixth, seventh and eighth grade tests…” before going on to ponder the implications of this iteration of the test (andreagabor.com). More than a month ago. The Post distributed copies and quoted excerpts of the 5th grade exam.

Ironically, it feels as if the only people who are studying the test and writing their responses to it are teachers and principals. Because this is a closed test, educators risk losing their jobs if they obtain and speak out about boot-legged copy of standardized tests. Still, it is possible for the world to hear the observations and concerns that educators have about the test. The day after the ELA, the organization I lead – the Teachers College Reading and Writing Project – opened a site (http://elafeedback.com) on which educators could post observations made during the test, and thoughts and concerns about the test. More than a thousand responses have been entered onto that site, and altogether, the responses show that teachers, principals and superintendents from both high achieving and high need schools have deep concerns about New York State’s test. Elafeedback.com gives a window into what educators are concerned about in regard to these tests. Now the question is – is anyone listening?

...

It is entirely likely that this test will be influential (even controlling) in decisions about how reading and writing are taught. It is especially likely that the exams will become curriculum in any state or city in which standardized tests have become deciding factors in whether teachers are hired or fired. In NYC, the scores that students receive on Pearson’s tests determine whether students have access to selective middle schools and secondary schools, allowing the tests to take on inordinate importance. On top of that, in NYC, teachers are ranked by name in newspapers based upon the scores their students receive, and this, again, means that the tests become an all-important measuring stick.

In the online posts at elafeedback.com, you’ll find a few issues that are raised again and again. One of these addresses the interpretation Pearson makes of close reading of nonfiction. For most teachers, the goal of teaching kids to read nonfiction successfully is to teach in such a way that students can learn from the nonfiction they read. That is, if they read an article on the Pony Express, the goal is that they learn quite a bit about that topic. If you look at the Common Core standards themselves, this reading work would encompass standards 1-3, which asks students to determine central ideas and supporting details, and analyze their development in the text, as well as standards 7-9, which asks students to synthesize and integrate, compare and contrast, and weigh and evaluate, ideas suggested by texts on the same subject.

Yet the Pearson exam seems to have asked few or no questions that addressed standards 7-9, as they chose to present students only with isolated texts rather than text sets, and many questions on standards 4-6, that ask students to analyze the craft and structure of texts. “Which term best describes the structure used in paragraphs 4-6?” “Why did the author include the image of….in line 12 of paragraph 5?” This sort of highly metacognitive, analytical reading-writing connection work is not usually the primary reading lens of nonfiction readers. Teachers are getting the message that their instruction should no longer channel students to read lots of nonfiction in order to expand their knowledge and grow ideas about a topic. Teachers are gathering that what counts to Pearson and New York State is that even children as young as nine year olds read nonfiction texts in order to analyze the author’s craft. This work has been propagated through the Publishers Criterion, a document offered by two authors of the Common Core, after the CCSS was ratified. As one poster, Sandra Wilde wrote, “ the items are constructed in a very narrow way, not from the standards themselves but from a narrow set of ideas - based on the Publisher's Guidelines”

I wish I had time to spend on this issue. It's too complex to discuss without doing your homework (which I haven't done), but I do have a few brief observations:

1. The reform movement has always been based on an odd alliance of that saw a chance to advance their generally well intended but otherwise disparate goals. One of these groups is theorists who wanted to try different pedagogical approaches and were running into resistance from the teaching establishment.

2. Though reform is often presented as a return to common sense basics, some of these pedagogical approaches, particularly those involving reading and writing, seem quite arcane. From first and second grade, a great deal of time is spent on fairly obscure concepts like the distinction between perspective and point of view.

3. If the tests we use to evaluate students, schools and teachers incorporate these ideas, then these approaches will become the standard.

and

4. While I don't want to dismissive of these approaches, I have a very different view. Kids are amazingly intelligent and inquisitive. If you give them the ability to read with little effort and good comprehension and you get them to read widely and deeply and to write clearly and thoughtfully, they will teach themselves better than you ever could.

Tech companies may lie to you but we never do

Item number 5 from The Worst Lies Tech Companies Tell You:

Cord cutters, pay attention: There is no such thing as an HDTV antenna. Or to put it another way, any TV antenna you've ever used, ever, is an HDTV antenna. Yes, all of them.

Check out the update on this post from 2010 or this post from 2011. (I've actually grown more suspicious of  some other aspects of that 2011 story, partly because the account conflicts with some things I've observed and partly because the New York Times has a record of serial inaccuracy on the OTA story.)

Jon Chait has to be pranking us?

I say this because of this review he posts of a book that seems deeply ahistorical:

The new right-wing book of the moment is American Betrayal, by Diana West, which offers the thesis that American foreign policy under presidents Roosevelt, Truman, and Eisenhower was secretly controlled by the Soviet Union. If this sounds like the sort of tract that would be written by somebody who thinks Joe McCarthy was absolutely right about everything, well, that’s exactly what Diana West does believe. She’s also quite the birther.

and

West argues not just that there were communist spies in the government or that American foreign policy failed to take a strong enough line, but that the Soviets controlled the American government in the way they controlled, say, East Germany. For instance, West argues that America provoked Japan into attacking Pearl Harbor — an American working for the Soviets “subverted relations between the US and Japan by inserting ‘ultimatum’ language into the cable flow that actually spurred the Japanese attack.”

So the downfall of the communist US government was due to John F Kennedy, lion of the right wing?  Who managed to squeak out a victory over Soviet pawn Rich Nixon (Eisenhower's Vice President, after all)?  Making JFK and Lyndon Johnson leading the only actual American led government in this 40 year period of subjugation?  Making the Great Society what a non-communist government would introduce in opposition to the communism of Richard Nixon? 

I am not saying that these things aren't possible in the sense of "logically possible".  But the Korean war and the Berlin blockage seem like odd policy decisions if the United States was a soviet satellite in the sense of East Germany.  Look what happened to Hungary in 1956 when it decided to buck Soviet control!  Now, this is completely separate from a claim that these presidents were rabid socialists or a competing strain of communist (kind of like China was).  It is the direct control from Moscow that seems to beg for a very high level of evidence.

So I guess I prefer to believe that Jon Chait got his facts wrong rather than actually imagining anybody would posit this theory of US history. 

The Epidemiology of Laughter

No, really. This post by famed sitcom writer/producer/director Ken Levine illustrate how infectious laughter follows some of the same rules as other infectious phenomena.

Comedy plays better in confined spaces. Laughs are louder when they don’t drift away.

Now you may say this is a superstition and I just want to be near that massage parlor, but (1) they don’t give group on’s, and (2) being in close quarters amplifies the laughter and laughter is infectious.

Whenever a sitcom episode goes into production the first order of business is a table reading. Several large tables are set up, the actors sit across from each other and read the script aloud as the writers and executives sit around them. Many shows I’ve worked on just hold their table readings right on their cavernous sound stages. On shows I’ve produced I insist we hold the table readings in conference rooms. Yes, it’s a little cramped, and chairs are pushed up against walls, but the difference in the reaction is startling. Laughs are so much bigger when you’re not at the bottom of the Grand Canyon.  Jokes are so much funnier when they don't echo.

Lest you think it’s just me, the table readings for CHEERS, FRASIER, WINGS, THE SIMPSONS, and BECKER were all held in conference rooms.

Levine also addresses the question of whether you should initially test a concept under overly friendly or overly hostile conditions.

Do we get an unfair reading as a result? Do the scripts appear funnier than they really are? Sometimes. There are producers who won’t change jokes later if it worked at the table reading. I’m not one of them. If a joke doesn’t work when it’s on its feet I cut it.  Table readings can always be deceiving.

But way more often, I’ve seen bad table readings done on the stage then gone back to the room and changed the shit out of the script. Later that day we'd have a runthrough of the original table draft and 70% of the stuff we planned to cut suddenly worked.

I’d rather err on the side of the table reading going well. Especially since you have the network and studio there as well. The less nervous they all are about the script, the better it is for all concerned.

The Totem Pole

Via Andrew Gelman, Gary Rubinstein has a very good post on what he observed while visiting a KIPP school in NYC (part of a larger series). I'll be getting back to the big issues Rubinstein raises soon, but first I wanted to take a moment on this side issue.

One thing that I see in schools a lot is the most experienced teachers getting to teach the high level class while the newbies have to teach the ninth grade remedial classes.  I suppose that this is some kind of reward for longevity, but it really is unfair.  If it really is about ‘the kids’ the new teachers should teach some of those easier classes.  I’m sorry to report that at KIPP they seemed to have the same sort of totem pole going on.  In the ninth grade wing is where I saw the most first year TFAers.

We hear a lot of complaints about how soft things supposedly are for long-time teachers, but the one place I've actually seen extensive evidence of this is an area that almost no one mentions (except, of course, for Rubinstein).

As a rule, first year teachers are best suited for advanced classes both in terms of strengths and weaknesses. New teachers tend to take a while to stop thinking in college terms. They are generally weak in classroom management and in those special communication skills needed for to handle lower level classes. Talking about concepts with peers and professors is very different from discussing those same concepts with bored ninth graders running behind grade level.

On the plus side, those concepts are fresher for just-out-of-college teachers and they inspire more passion. New teachers generally have more energy and are better able to keep up with classes of academically advanced kids. On top of that, college bound students often respond better to teachers they see as part of the college experience..

There is, as always, a context of bigger issues here, but for now, if you're looking for a reform to get behind, you could do worse than  tearing down that totem pole.

At this point, seeing even inaccurate coverage of OTA television seems like a triumph

This Marketplace story does leave a bit to be desired in terms of accuracy and completeness. In the audio version, the reporter, Sally Herships, like almost everyone covering this story, seems confused by digital versus amplified antennas and TV antennas versus satellite leading her to greatly overstate the price range of the equipment -- think under $10 if you don't need amplification (if not cheaper) -- and she failed to mention that the picture quality is actually better for free TV or that, in the region being discussed, you can get close to one hundred channels over the air or that with an adapter you can watch and digitally record those channels on you laptop.

All of that aside, this is still a story about doing without cable (or at least certain channels on cable), airing on one of the biggest and most respected financial news programs and it lists rabbit ears as the first option. I've been following this story closely for years now and I honestly can't recall this happening before. There have been a few voices in the wilderness on the subject -- Rajiv Sethi being perhaps the first and most notable -- but that was the blogosphere. A story like this in mainstream media is a very promising development, particularly given that the last time I heard the topic broached on Marketplace it was immediately dismissed by their guest expert (from the NYT, of course) who claimed that you can only get "a handful of channels" with an antenna in LA.

The story digital broadcasting has always come down to the question of whether the superior technology could establish a foothold before regulatory capture and the huge imbalance of hype drove it out of business. In that context, this is a good day for the medium.

Metablogging -- stag hunts, misalignment, principal agents and all that jazz

Shortly after Josh Marshall posted this analysis of recent events in the GOP, Joseph called me up to compare reactions. We've been having this conversation for so many years that much of it has devolved into a self-referential shorthand. As an illustration, at one point, after a fairly long-winded comment by me, Joseph simply said "Stag hunt." and tossed the ball back to me and we moved on to the next topic.

We agreed (with, I assume, fingers crossed on both ends of the line) that we'd write some posts on the subject, but I'm starting to think that it might be more useful to step back for a minute and talk about how we've been framing the question of what's going on in the Republican Party (politically, not socially or in terms of policy. Those are entirely different metaposts).

For years now, the two of us have been talking about the post-Tea Party GOP in terms of a multi-player stag hunt. Over the past few years the stakes (particularly the costs of failure) have increased. At the same time, participation rates required to take down the stag have also increased. As a result, progressively smaller groups have gained the power to kill the enterprise. (In a different conversation Joseph pointed out that, in a military context, shooting deserters is also a predictable result of this situation.) We could dig deeper into examples and implications (particularly with respect to the trade off between the power of an alliance vs. its stability) but for now I want to limit the discussion to framing.

(there might also be a place here to talk about symmetry breaking, but I'd need to give that some thought first.)

Another way of looking at the story we've found useful is to look at misalignment of interests, especially what looks to us two non-economists as a particularly nasty two-level principal agent problem where a small group of big donors determine the pool of viable candidates and a relatively small but coherent subgroup of the primary voters make the purchasing decisions for the entire party. You'll notice that, like the stag hunt frame, under this scenario small groups can acquire disproportionate power.

And of course there's the mandatory Influence reference, framing the story in terms of social psychology. If you check out the chapters on commitment and consistency, social proof, and scarcity you'll find all sorts of applicable discussions of the ways groups united by a common belief system deal with ideological challenges and the loss of dominance.

Nothing particularly fresh or profound here, but these idea have proven a pretty good framework recently. I'm not saying they should be the basis of the standard narrative -- I'm not sure there should be a standard narrative -- but they do come in handy. More importantly, I think you can make the case that too little of the public discourse is spent examining underlying assumptions and asking about the different ways to frame our questions.

The real challenge of Libertarian populism

According to Wikipedia, the night-watchman state (a foundation of Libertarianism) is defined as:

A night-watchman state, or a minimal state, is variously defined by sources. In the strictest sense, it is a form of government in political philosophy where the state's only legitimate function is the protection of individuals from assault, theft, breach of contract, and fraud, and the only legitimate governmental institutions are the military, police, and courts.

The bolding was added by me.

I think the real challenge of Libertarianism is not issues like Libertarian populism but rather the requirement that a Libertarian state protect against fraud.  I think that this is a more important the weaker that government gets, the more critical each function becomes.  Yet modern corporations are increasingly being allowed to get away with Fraud (and with changing of contracts).  These are huge problems because if you are going to enforce contracts against the politically weak, you also need to enforce them against the politically strong or you really do get feudalism. 

Trials like that of the S&P ratings schemes (where they argued ratings were puffery) are critical because it may be the case that a reasonable person might well think that misrepresentation was a form of fraud.  Saying that "lies=marketing" seems to be a rather obvious attempt to evade the antifraud portion of the states function. 

In the same sense, changing the rules under which a corporation operates can be an extremely dodgy move.  Changing the rules can be okay, but the optics are terrible when a major corporation shifts the rules when an initial attempt to do something fails.  It doesn't matter that the main victim is Carl Icahn. who is hardly at risk of poverty. 

To me, this is really the central issue of the modern Libertarian approach.  It isn't impossible that you could have a Libertarian populism, but I think the backbone of this approach would have to be that the same rules apply to everyone (even if they have a lot of money). 

Price discrimination at Ralph's

This is a new one on me.

I was in the grocery store last night when the thought hit me that I hadn't had a root beer float in a long time, too long in fact, but as I walked back and forth along the soda aisle I could not find any cheap root beer. There were various Coke and Pepsi products but to get a reasonable price you had to buy four (I don't drink much soda -- eight liters is a bit much). There were also expensive specialty brands, but no house brands. Annoyed, I grabbed a bottle of A&W and went on with my shopping only to see the store brands I was looking for one aisle over mixed in rather incongruously with the sports drinks.

Normally you'd expect to see the house brands near the corresponding name brand products, but I suspect that the previously mentioned "buy four" promotions prompted the shift, that given the choice between paying the inflated list price or dealing with the inconvenience of buying far more than they needed, too many customers were saying to hell with it and grabbing a Shasta or a Big K which had no restrictions and was even cheaper than the discounted name brands.

Given the competition from places like Wal-Mart and Food-4-Less and the 99 cents store, stocking cheap store brands can keep price sensitive customers coming back but making those brands hard to find can keep the less price sensitive customers spending more than they have to.

Worst Statistic of the Day

Via Lawyers, Guns, and Money comes the piece most detached from reality I've seen this year:

When Josh McFarland graduated from Stanford he owed $40,000 in student loans and couldn't fathom a way he'd ever pay it off and have a future for himself - not unusual for the typical young adult these days. Then he went to work for Google.

As a product manager, he got stock options and cashed them in over the five years he worked there. He married a fellow Google employee, so she had stock too. Then she moved on to Yelp, and he quit to launch TellApart, which provides technology solutions for e-commerce sites. Now 33, McFarland has a 3-year-old and a newborn and no longer has to think about his student loan: His company has $17.75 million in venture capital investment. While he doesn't consider himself retire-now rich, his piece of the company affords him what he calls "breathing room" and what other people might call wealth.

McFarland is on the starting end of Generation Y, the cohort born in the United States after 1980 that is typically portrayed as saddled with massive student debt, underemployed and underpaid. More than a third of the 80 million group of so-called millennials live with their parents, according to the Pew Research Group.

But McFarland is part of the sizeable minority that is doing quite well: 12 million Gen Y-ers make more than $100,000, according to the Ipsos MediaCT's Mendelsohn Affluent Survey. Many of them, in technology fields, live frugal work-based lifestyles and are not saddled with the six-digit student debt held by doctors and lawyers.

Paul Campos researched this statistic and discovered where it came from:

The source of the wrongheaded statistic appears to be this: the cited survey claims 59 million Americans live in households with incomes of $100,000 plus, and that 20% of “affluent consumers” are in the Gen Y cohort (this appears to mean that 20% of the adults who live in households with $100,000+ incomes are of this age). The story’s author then did some very bad math to generate the claim that “12 million Gen Y-ers make more than $100,000.” 

But that is totally different than these people making > $100,000 per annum.  That could also be people who are utterly unemployable living in their parent's basement. Or a trophy wife/husband of a hedge-fund billionaire. Or it could be 5 young adults sharing a house together, all making $22,000 a year as Starbucks' employees. 

At the very least it uses a highly no-representative sample and portrays the wealth of young adults in a completely misleading way.  This does not add to the conversation and it appears in the New York Times.  If only they had a prominent economist on staff who could consult over this type of data . . .

Weekend blogging -- Patents and product testing

Of course, these days I'm sure someone has patented the general idea of making a death ray. Perhaps that's an upside of having trolls.

"An astonishing act of statistical chutzpah" -- details on the Tony Bennett scandal

Jordan Ellenberg at Slate and Anne Hyslop at the New America Foundation have posts up explaining exactly how the Indiana Department of Education managed to change the grade of an influential donor's charter school from a C to an A. It appears to be your basic DATA COOKING 101. After seeing results you don't like, go back, try different weightings and look for excuses to drop bad scores then apply these changes selectively.

The 'selectively' part is particularly important and hasn't gotten the attention it deserves:

Two Indianapolis Public Schools might never have been taken over by the state if then-Superintendent of Public Instruction Tony Bennett had offered the district the same flexibility he granted a year later to the Christel House Academy charter school.

The issue was similar in both cases. Christel House had recently added ninth and 10th grades, and IPS’ Howe and Arlington had added middle school grades. The students who filled those seats posted poor enough scores to drag down the schools’ overall ratings.

In the case of Christel House, emails unearthed by The Associated Press show Bennett’s staff sprung into action in 2012 when it appeared scores from the recently added grades could sink the highly regarded school’s rating from an A to a C. Ultimately, the high school scores were excluded and the school’s grade remained an A.

But in 2011, after IPS’ then-Superintendent Eugene White demanded Bennett consider the test scores of high school students separately from those of middle school students so the high schools could avoid state takeover, Bennett was unmoved.

As for the specifics, Hyslop goes into more detail (and has a wonderfully apt movie quote to make her point), but Ellenberg probably does the better job summing it up:

Here’s where Bennett’s team found the loophole big enough to drive a charter school through. A normal person would do exactly what Chief Accountability Officer Jon Gubera did—give Christel the weighted average of its elementary/middle school score, according to the rules for elementary/middle schools, and its high school score, according to the rules for high schools.

But Bennett had a better idea. Christel was, technically speaking, not a high school, so the statutory formula for the high school grades didn’t apply. But it also didn’t have all four high school measures, so, he argued, the rules for combined schools didn’t apply either. There were just 13 schools in the state that had both middle school and high school grades but no seniors. For these schools, Bennett reasoned, the Indiana education poobahs should have a free hand to set the grades however they pleased. You can guess what happened next: Bennett ruled that the ninth- and 10th-graders in these schools didn’t count at all. So it was that the offending algebra grades vanished in a puff of bureaucratic smoke...

This was an act of astonishing statistical chutzpah. Suppose the syllabus for my math class said that the final grade would be determined by averaging the homework grade and the exam grade, and that the exam grade was itself the average of the grades on the three tests I gave. Now imagine a student gets a B on the homework, gets a D-minus on the first two tests, and misses the third. She then comes to me and says, “Professor, your syllabus says the exam component of the grade is the average of my grade on the three tests—but I only took two tests, so that line of the syllabus doesn’t apply to my special case, and the only fair thing is to drop the entire exam component and give me a B for the course.”

Ellenberg then made an observation that echoed some of my earlier points about how the mindset of the reform movement can enable these ethical lapses:

The saddest part is that I’m guessing Bennett sincerely felt he was doing the right thing. In his mind, he knew Christel was a great school, so if the scores said otherwise, the scores had to be wrong. In this respect, ironically, he ends up echoing his policy opponents, adopting the position that a mechanistic testing and scoring procedure can’t be allowed to override firsthand knowledge about teachers and schools.

The Op-ed no one wanted to print

When it comes to the education beat, this guy has one hell of a resume.

John Merrow began his career as an education reporter with National Public Radio in 1974, with the weekly series, “Options in Education.” In 1984, Merrow branched out into public television. He served as host of The Merrow Report, an award-winning documentary series, and currently is the Education Correspondent for PBS NewsHour. Merrow’s work has taken him from community colleges to kindergarten classrooms, from the front lines of teacher protests to policy debates on Capitol Hill. His varied reporting has continually been on the forefront of education journalism.

Of course, all of these qualifications don't necessarily make a person right, but they generally do make a person publishable. That's what makes this story curious, because Merrow can't seem to get anyone to publish the following op-ed. This is the second time in a row that he's found newspapers and magazines reluctant to publish reporting on Micelle Rhee.

As mentioned before, Rhee has gone from feted to indefensible so quickly that it's difficult for most journalists and pundits to cover her current activities without looking comically gullible for swallowing her previous rhetoric. (And journalists don't like looking gullible.)

CAVEAT EMPTOR: MICHELLE RHEE’S EDUCATION REFORM CAMPAIGN

"Today, too many of America’s children are not getting the quality education they need and deserve. StudentsFirst is helping to change that with common sense reforms that help make sure all students have great schools and great teachers." (StudentsFirst press release, emphasis added)

Michelle Rhee created StudentsFirst after leaving her post as Chancellor of Washington, DC’s Public Schools in the fall of 2010. She announced her intentions on “Oprah” that December: to fix America’s schools by enrolling one million members and raising one billion dollars.[2]

Easily America’s most visible education activist, she has been crisscrossing the country lobbying for change and donating money to candidates whose policies she supports. StudentsFirst claims to have helped pass 110 ‘student-centered policies’ in 18 states.

Because Ms. Rhee is trying to persuade the rest of the country to do as she did in Washington, it’s worth asking what her ‘common sense reforms’ accomplished when she had free rein to do as she wished.

She was definitely in charge. Her boss, a popular new mayor, told his Cabinet that trying to block his Chancellor was a firing offense.  The business community, a public fed up with school failure, and the editorial pages of The Washington Post were enthusiastic supporters. Moreover, she had virtually no opposition: the local school board had been abolished when the Mayor took over, and the teachers union, reeling from its own financial scandals, had an untested rookie president. She knew how lucky she was.

"I’m living what I think education reformers and parents throughout this country have long hoped for, which is, somebody will just come in and do the things that they felt was in the best interest of children and everything else be damned. (Interview, fall 2007)"

She lived that dream for 40 months.  She opened schools on time, added social workers, beefed up art, music and physical education, and dramatically expanded preschool programs.  The latter may represent her greatest success, because children who began their schooling in the expanded preschool program tend to do well on the system’s standardized test in later years.

Ms. Rhee made her school principals sign written guarantees of test score increases. It was “Produce or Else” for teachers too. In her new system, up to 50% of a teacher’s rating was based on test scores, allowing her to fire teachers who didn’t measure up, regardless of tenure.  To date, nearly 600 teachers have been fired, most because of poor performance ratings. She also cut freely elsewhere–closing more than two-dozen schools and firing 15% of her central office staff and 90 principals.

When Ms. Rhee departed in October 2010, her deputy, Kaya Henderson, took over. She has stayed the course for the most part, although test scores now make up–at most–35% of a teacher’s rating score.

Some of the bloom came off the rose in March 2011 when USA Today reported on a rash of ‘wrong-to-right’ erasures on standardized tests and the Chancellor’s reluctance to investigate.  With subsequent tightened test security, Rhee’s dramatic test scores gains have all but disappeared. Consider Aiton Elementary: The year before Ms. Rhee arrived, 18% of Aiton students scored proficient in math and 31% in reading. Scores soared to nearly 60% on her watch, but by 2012 both reading and math scores had plunged more than 40 percentile points.

But it’s not just the test scores that have gone down. Six years after Michelle Rhee rode into town, the public schools seem to be worse off by almost every conceivable measure.

For teachers, DCPS has become a revolving door. Half of all newly hired teachers (both rookies and experienced teachers) leave within two years; by contrast, the national average is understood to be between three and five years. Veterans haven’t stuck around either. After just two years of Rhee’s reforms, 33% of all teachers on the payroll departed; after 4 years, 52% left.

It has been a revolving door for principals as well.  Ms. Rhee appointed 91 principals in her three years as chancellor, 39 of whom no longer held those jobs in August 2010. Some chose to leave; others, on one-year contracts, were fired for not producing quickly enough.  Several schools are reported to have had three principals in three years.

Child psychiatrists have long known that, to succeed, children need stability.  Because many of the District’s children face multiple stresses at home and in their neighborhoods, schools are often that rock. However, in Ms. Rhee’s tumultuous reign, thousands of students attended schools where teachers and principals were essentially interchangeable parts, a situation that must have contributed to the instability rather than alleviating it.

Although Ms. Rhee removed about 100 central office personnel in her first year, the central office today is considerably larger, with more administrators per teachers than any of the districts surrounding DC.  In fact, the surrounding districts reduced their central office staff, while DC’s grew.  The greatest growth in DCPS over the years has been in the number of central office employees making $100,000 or more per year, from 35 when she arrived to 99 at last count.

Per pupil expenditures have gone up sharply, from $13,830 per student to $17,574, an increase of 27%, compared to 10% inflation in the Washington-Baltimore region. So have teacher salaries; DC teachers now earn on average more than their counterparts in nearby districts in Virginia and Maryland.

Enrollment declined on Ms. Rhee’s watch and has continued under Ms. Henderson, as families continue to enroll their children in charter schools or move to the suburbs.  The year before she arrived, DCPS had 52,191 students. In school year 2012-13 it enrolled about 45,000, a loss of roughly 13%.

Even students who have remained seem to be voting with their feet, because truancy in DC is a “crisis” situation, and Washington’s high school graduation rate is the lowest in the nation.  The truancy epidemic may be the most telling data point of all, because if young people in this economy are not going to school, something is very wrong. They are not skipping school to work–because there are no jobs for unskilled youth.

Ms. Rhee and her admirers point to increases on the National Assessment of Educational Progress, an exam given every two years to a sample of students under the tightest possible security.  And while NAEP scores did go up, they rose in roughly the same amount as they had under her two immediate predecessors, and Washington remains at or near the bottom on that national measure.

The most disturbing effect of Ms. Rhee’s reform effort is the widening gap in academic performance between low-income and upper-income students, a meaningful statistic in Washington, where race and income are highly correlated.  On the most recent NAEP test (2011) only about 10% of low-income students in grades 4 and 8 scored ‘proficient’ in reading and math. Since 2007, the performance gap has increased by 29 percentile points in 8th grade reading, by 44 in 4th grade reading, by 45 in 8th grade math, and by 72 in 4th grade math. Although these numbers are also influenced by changes in high- and low-income populations, the gaps are so extreme that is seems clear that low-income students, most of them African-American, generally did not fare well during Ms. Rhee’s time in Washington.

English Language Learners in Washington’s schools are also struggling. Title III of ESEA requires progress on three distinct measures: progress, attainment and what ‘No Child Left Behind’ calls ‘adequate yearly progress.’  DC failed on two out of three last year.

DC doesn’t fare well in national comparisons either.  Between 2005 and 2011, black 8th graders in large urban districts gained five points in reading, while their DCPS counterparts lost two points, according to a study by the DC Institute of Public Policy released this spring. Between 2005 and 2011 in large, urban districts, Hispanic eighth-graders gained six points in reading (from 243 to 249), black eighth-graders gained five points (from 240 to 245), and white eighth-graders gained three points (from 270 to 273). In District of Columbia Public Schools, however, Hispanic eighth-graders’ scores fell 15 points (from 247 to 232), black eighth-graders’ scores fell two points (from 233 to 231), and white eighth-graders’ scores fell 13 points (from 303 to 290).

The states that have adopted her approach, and others now being lobbied, might want to make their own data-driven decisions.

A final (?) news wrap-up on the Bennett story

As previously mentioned:

INDIANAPOLIS (AP) — Former Indiana and current Florida schools chief Tony Bennett built his national star by promising to hold “failing” schools accountable. But when it appeared an Indianapolis charter school run by a prominent Republican donor might receive a poor grade, Bennett’s education team frantically overhauled his signature “A-F” school grading system to improve the school’s marks.

You can get Joseph's reaction here and mine here and here.

Despite considerable support from the reform movement, you can add another 'former' to that paragraph.

Bennett said he resigned “because I don’t believe it would be fair to be distracted” by what he characterized as “malicious and unfounded” reports. 

Just yesterday, Gov. Scott told Channel 5 in West Palm Beach that Bennett is “doing a great job.

In what was already a bad news day for Bennett:

In June of 2011, Tony Bennett, then Indiana’s superintendent of public instruction, picked a for-profit education company in Florida to run a group of Indianapolis public schools.

The company, Charter Schools USA, set up operations in Indianapolis soon after the announcement and officially began running Manual High School, T.C. Howe High School and Emma Donnan middle school in the late summer of 2012. Millions of Indiana tax dollars have since flowed to the company, which has received many good reviews for its work in Indianapolis.

But a recent hiring decision by Charter Schools USA is sure to raise eyebrows and questions about conflicts of interest, particularly now that Bennett is embroiled in a massive controversy centering on special treatment given to certain Indiana schools during his tenure.

The decision: Charter Schools USA earlier this year hired Tony Bennett’s wife, Tina, as a regional director based in Florida, where Tony Bennett was hired late last year as commissioner of education. And, so, the bottom line is this: Tina Bennett is now earning a paycheck from the company her husband hand-picked to take over schools in Indiana, a decision that was very good for the company’s financial fortunes.

It’s important to note that Tina Bennett is a longtime educator, a former school administrator and counselor. She is also an advocate of the type of school choice efforts that Charter Schools USA is built on. In Indiana, she faced criticism and sometimes cruel treatment for taking a job with education groups tied to her husband’s former office. But it’s understandable that she would seek work in the education field.

To provide some context (and a bit of schadenfreude) for Bennett's fall, here's a reminder of where Bennett ranked in the reform firmament.

Value added testing without a gold standard outcome

From a Megan Pledger comment on StatChat comes this paper (pdf) on value added testing models for evaluating teachers.  The following concerns were brought up:

In the real world of schools, data is frequently missing or corrupt. What if students are missing past test data? What if past data was recorded incorrectly (not rare in schools)? What if students transferred into the school from outside the system?

The modern classroom is more variable than people imagine. What if students are team-taught? How do you apportion credit or blame among various teachers? Do teachers in one class (say mathematics) affect the learning in another (say science)?

Every mathematical model in sociology has to make rules, and they sometimes seem arbitrary. For example, what if students move into a class during the year? (Rule: Include them if they are in class for 150 or more days.) What if we only have a couple years of test data, or possibly more than five years? (Rule: The range three to five years is fixed for all models.) What’s the rationale for these kinds of rules?

Class sizes differ in modern schools, and the nature of the model means there will be more variability for small classes. (Think of a class of one student.) Adjusting for this will necessarily drive teacher effects for small classes toward the mean. How does one adjust sensibly?

While the basic idea underlying value-added models is the same, there are in fact many models. Do different models applied to the same data sets produce the same results? Are value-added models “robust”?

Since models are applied to longitudinal data sequentially, it is essential to ask whether the results are consistent year to year. Are the computed teacher effects comparable over successive years for individual teachers? Are value-added models “consistent”?

A lot of these concerns have been independently voiced by Mark P.  However, what is especially concerning is the idea that we could iterate through these assumptions to find a school ranking that satisfies some prior.  This can be good under some circumstances -- Thomas Lumley gives an example of a model that clearly mixed up rankings of some kind of sports team (this isn't my area of expertise so I apologize that I don't recognize the teams or sport involved).  But it does show how difficult these models are, even with the best faith involved.  Still, in the case of Dr. Lumley's example there is a universal outcome that has broad agreement (does this team win games) that is being predicted.  In education we lack this very clean outcome which is where it gets tricky -- in a sense we are modeling a latent variable (student outcomes). 

All of this suggests that we should be cautious about these models and perhaps this would be an appropriate time to put some serious effort into student outcomes ascertainment so that it will be easier to calibrate these statistical models (making the outcome the test score seems clever but merely hides the problem rather than solving it unless we are confident that the score is a very good measure of outcomes).