The Annotated "Evaluating New York Teachers, Perhaps the Numbers Do Lie"

As promised, here are some comments (in brackets) on Michael Winerip's NYT article on the city's teacher evaluation process.

Last year, when Ms. Isaacson was on maternity leave, she came in one full day a week for the entire school year for no pay and taught a peer leadership class.

...

[One thing that Winerip fails to emphasize (though I suspect he is aware of it) is how common stories like this are. Education journalists often portray ordinary excellence as something exceptional. This is partly due to journalistic laziness -- it's easier to describe something as exceptional than to find something that actually is exceptional -- and partly due to the appeal of standard narratives, in this case the Madonna/whore portrayal of teachers (I would used a non-gender specific analogy but I couldn't come up with one that fit as well.)]

The Lab School has selective admissions, and Ms. Isaacson’s students have excelled. Her first year teaching, 65 of 66 scored proficient on the state language arts test, meaning they got 3’s or 4’s; only one scored below grade level with a 2. More than two dozen students from her first two years teaching have gone on to Stuyvesant High School or Bronx High School of Science, the city’s most competitive high schools.

...

[Everything in this article inclines me to believe that Ms. Isaacson is a good teacher but we need to note that this is a fairly easy gig compared to other urban schools, particularly for someone with her background. Students at places like the Lab School tend to be more respectful and attentive toward academically successful people like Ms. Isaacson. In many schools, this can actually make students initially distrustful.]

You would think the Department of Education would want to replicate Ms. Isaacson — who has degrees from the University of Pennsylvania and Columbia — and sprinkle Ms. Isaacsons all over town. Instead, the department’s accountability experts have developed a complex formula to calculate how much academic progress a teacher’s students make in a year — the teacher’s value-added score — and that formula indicates that Ms. Isaacson is one of the city’s worst teachers.

According to the formula, Ms. Isaacson ranks in the 7th percentile among her teaching peers — meaning 93 per cent are better.

[One of the fallacies that follow from this Madonna/whore narrative is the idea that, since you have such a clearly bi-modal distribution, any metric that's correlated with teaching quality should be able to winnow the good from the bad. In reality you have a normal distribution with noisy data and a metric that doesn't correlate that well. The result, unsurprisingly, is a large number of teachers apparently misclassified. What is surprising is that more people didn't foresee this fairly obvious outcome.]

This may seem disconnected from reality, but it has real ramifications. Because of her 7th percentile, Ms. Isaacson was told in February that it was virtually certain that she would not be getting tenure this year. “My principal said that given the opportunity, she would advocate for me,” Ms. Isaacson said. “But she said don’t get your hopes up, with a 7th percentile, there wasn’t much she could do.”

That’s not the only problem Ms. Isaacson’s 7th percentile has caused. If the mayor and governor have their way, and layoffs are no longer based on seniority but instead are based on the city’s formulas that scientifically identify good teachers, Ms. Isaacson is pretty sure she’d be cooked.

[Well, as long as it's scientific.]

She may leave anyway. She is 33 and had a successful career in advertising and finance before taking the teaching job, at half the pay.

...
[This isn't unusual. I doubled my salary when I went from teaching to a corporate job. Plus I worked fewer hours and they gave us free candy, coffee and the occasional golfing trip.]
...

The calculation for Ms. Isaacson’s 3.69 predicted score is even more daunting. It is based on 32 variables — including whether a student was “retained in grade before pretest year” and whether a student is “new to city in pretest or post-test year.”

Those 32 variables are plugged into a statistical model that looks like one of those equations that in “Good Will Hunting” only Matt Damon was capable of solving.

The process appears transparent, but it is clear as mud, even for smart lay people like teachers, principals and — I hesitate to say this — journalists.

[There are two things about this that trouble me: the first is that Winerip doesn't seem to understand fairly simple linear regression; the second is that he doesn't seem to realize that the formula given here is actually far too simple to do the job.]


Ms. Isaacson may have two Ivy League degrees, but she is lost. “I find this impossible to understand,” she said.

In plain English, Ms. Isaacson’s best guess about what the department is trying to tell her is: Even though 65 of her 66 students scored proficient on the state test, more of her 3s should have been 4s.

But that is only a guess.

[At the risk of being harsh, grading on a curve should not be that difficult a concept.]

Moreover, as the city indicates on the data reports, there is a large margin of error. So Ms. Isaacson’s 7th percentile could actually be as low as zero or as high as the 52nd percentile — a score that could have earned her tenure.

[Once again, many people saw this coming. Joel Klein and company chose to push forward with the plan, even in the face of results like these. Klein has built a career largely on calls for greater accountability and has done very well for himself in no small part because he hasn't been held accountable for his own record.]

I've left quite a bit out so you should definitely read the whole thing. It's an interesting story but if anything here surprises you, you haven't been paying attention.

The lessons of Motown

Michael Winerip is a much better than average education reporter. He doesn't have a great grasp of the numbers or of the implications of the policies, limitations which have kept him from getting a jump on the story the way, say, This American Life did with the financial crisis, but he has kept up with it while most of his colleagues are still reporting discredited narratives from interested parties.

This article on Detroit is a good example. He doesn't connect some important dots but he does a good job reporting what he sees. (you'll find my comments in brackets):

In 2009, Detroit public schools had the lowest scores ever recorded in the 21-year history of the national math proficiency test.

The district had a budget deficit of $200 million.

About 8,000 students were leaving Detroit schools each year.

Political leaders had to do something, so they rounded up the usual whipping boys:

Wasteful bureaucrats. In 2009, the governor appointed an emergency financial manager, Robert Bobb, a former president of the Washington school board, to run the Detroit district. Mr. Bobb is known nationally for his work in school finance, and recruiting him took a big salary, $425,000 a year. He has spent millions more on financial consultants to clean up the fiscal mess left by previous superintendents.

[A large number of people are acquiring a great deal of money and power through the reform movement. This doesn't mean that these people don't have good intentions or that they are not worth the money they're being paid but it does mean that this group, which includes high profile figures like Joel Klein and Michele Rhee, has a vested interest in these policies. It also means that when Michele Rhee brags about being a counterbalance to the special interests, she's not being entirely honest.]

Greedy unions. Though Detroit teachers make considerably less than nearby suburban teachers (a $73,700 maximum versus $97,700 in Troy), Mr. Bobb pressed for concessions. He got teachers to defer $5,000 a year in pay and contribute more for their health insurance. Last week, the Republican-controlled Legislature approved a bill to give emergency managers power to void public workers’ contracts. If signed by the governor, Mr. Bobb could terminate the Detroit teachers’ union contract.

Traditional public schools full of incompetent veteran teachers. Michigan was one of the first states to embrace charter schools, 15 years ago. Currently there are as many Detroit children in charters — 71,000 — as in district schools. Now there is talk of converting the entire Detroit district (which is 95 percent African-American) to charters. Supporters say this could generate significant savings, since charters are typically nonunion and can hire young teachers, pay them less and give them no pensions.

[Before we go on, this would seem to be an almost perfect test of the large-scale charter school model (as compared to the more limited role I've advocated). Charter schools have been put forward as the solution for this very kind of troubled urban district.]

So now, two years later, how are the so-called reforms coming along?

Not great.

Since Mr. Bobb arrived, the $200 million deficit has risen to $327 million. While he has made substantial cuts to save money — including $16 million by firing hundreds of administrators [Of course, he's spent millions making those cuts] — any gains have been overshadowed by the exodus of the 8,000 students a year. For each student who departs, $7,300 in state money gets subtracted from the Detroit budget — an annual loss of $58.4 million.

[Economic conditions and demographic shifts still trump any educational reform proposed so far. People need to remember this.]

Nor have charters been the answer. Charter school students score about the same on state tests as Detroit district students, even though charters have fewer special education students (8 percent versus 17 percent in the district) and fewer poor children (65 percent get subsidized lunches versus 82 percent at district schools). It’s hard to know whether children are better off under these “reforms” or they’re just being moved around more.

[As mentioned before, there are a number of possible biasing effects (peer, placebo, Hawthorne, selective attrition, etc.) that may be inflating the charter's scores. In other words, they are not outperforming the public schools and they may be doing much worse.]

Steve Wasko, public relations director for Mr. Bobb and the Detroit schools, did not respond to a dozen voice mails and e-mails seeking comment. Those who know Mr. Wasko say he cares about Detroit and is sick of the national media portraying the city as hopeless.

[You have a public relations director who can't work a talk with the New York Times into his schedule. This alone raises questions about the Bobb administration. It also suggests some other options for cost cutting (who do you think makes more, a starting teacher or a public relations director?)]

...

Last spring, Mr. Bobb had planned to close 50 schools with dwindling enrollment. But his list was reduced to 30 after several public meetings at which parents and staff members pleaded their school’s case before the all-powerful Mr. Bobb.

In June, Mr. Bobb held a news conference at Carstens Elementary — one of the schools spared — to announce the 30 closings.

One reason Carstens survived was an article in The Detroit Free Press last March headlined “Carstens Elementary on DPS closing list is a beacon of hope.”

The school, surrounded by vacant lots and abandoned houses, serves some of the city’s poorest children. Thieves who broke into the school last year escaped by disappearing into what the police call “the woods” — the blocks and blocks of vacant houses.

Yet Carstens students perform well on state tests, repeatedly meeting the federal standard for adequate yearly progress.

[As seen before, good teachers and schools often end up bearing the brunt of our current crop of reforms.]

“We try to fill in the holes in our children’s lives,” said Rebecca Kelly-Gavrilovich, a Carstens teacher with 25 years’ experience. Students get free breakfast, lunch and — if they attend the after-school program — dinner.

To have more money for instruction, teachers sit with students at lunch, saving the school from having to hire lunchroom aides. Teachers hold jacket and shoe drives for children who have no winter coats and come to school in slippers. At Thanksgiving every child goes home with a frozen turkey donated by a local businessman. Twice a year a bus carrying a portable dentist’s office arrives, and a clinic is set up at the school so children can get their teeth checked.

Despite all this, teachers worry that Carstens’s appearance on Mr. Bobb’s closing list — even though it was brief — means the end is near. Anticipating the worst, several parents have taken their children out of Carstens, enrolling them elsewhere, including at charters and suburban schools.

Carstens’s enrollment is half of what it was a few years ago. Every hallway has empty classrooms, giving the school a desolate feeling.

Mr. Bobb has set off a vicious cycle undermining even good schools. The more schools he closes to save money, the more parents grow discouraged and pull their children out. The fewer the children, the less the state aid, so Mr. Bobb closes more schools.

[This is a pattern we're seen before. Check Ravitch for specifics.]

Carstens has also been harmed by poor personnel decisions made by the district. Last year, 1,200 teachers took the retirement buyout, and Mr. Bobb laid off 2,000 others in the spring. Then in the fall, he realized he needed to hire the 2,000 back, and chaos ensued.

[Also something we've been warning about.]

At Carstens, a kindergarten class of 30 had no teacher until October; teachers at the school took turns supervising the class. “How do you think parents feel when there’s a different teacher every day?” said Mike Fesik, the current teacher.

It’s hard to understand why any teacher who could leave Detroit stays, but they do. Kim Kyff, with 22 years’ experience, is one of the lead teachers at Palmer Park, the elementary and middle school that opened last fall. In 2007 she was the Michigan teacher of the year. She has had offers from suburban schools, but stays because she believes that in Detroit, she has a better shot at being a beacon of hope.

[I should say more but it's late and I'm already depressed as hell. It's times like these when I have trouble not believing that, not only do we not care about children, we actually go out of our way to screw over those who do care. (yes, that's a lot of 'not's but given the hour what do you expect?)]

Does anyone out there know Michael Winerip?

If so, please tell him that he needs to read this discussion on Andrew Gelman's blog. There are things he that he really ought to know.

In other words, Good Will Hunting understood sigma notation

From Michael Winerip's NYT article:

The calculation for Ms. Isaacson’s 3.69 predicted score is even more daunting. It is based on 32 variables — including whether a student was “retained in grade before pretest year” and whether a student is “new to city in pretest or post-test year.”

Those 32 variables are plugged into a statistical model that looks like one of those equations that in “Good Will Hunting” only Matt Damon was capable of solving.

I'll have more to say about this later. I don't think Winerip really understands what's going on here but the story's definitely worth a read.

Update: it is now later.