Success is not a scalar

I have been meaning to blog about this story for a long time. It raises all sorts of interesting and important questions about statistics and public policy that could merit any number of posts. Unfortunately, I haven't found bandwidth to do the topic justice. I certainly don't have time now but things are starting to heat up with the Success Academies and I need to get something down on paper, even if it is just a few quick bullet points.

There is a long list of assumptions and potential failure points associated with the system of metrics we use to evaluate schools;

One of the most troubling signs of a breakdown in the system is a divergence between the primary metric you are using for ranking and other metrics that should be measuring something similar;

Especially worrisome is the use of low stakes/high-stakes tests. These are tests where the outcomes have relatively little direct impact on the students who take them but have a large impact on schools and administrators. By comparison, the SAT, the ACT, the GRE, etc. are high-stakes for the students taking them;

It is particularly worrisome when the students from an institution do particularly well on a low stakes/high-stakes test then do much worse on a high-stakes/high-stakes test.

With that in mind, check out this from Juan Gonzalez writing for the New York Daily News:

Days earlier, Moskowitz had stunned many in this town by asking the state to grant her 14 new charter schools, thus potentially catapulting her network to 46 schools.

The first Success graduating class, for example, had just 32 students. When they started first grade in August 2006, those pupils were among 73 enrolled at the school. That means less than half the original group reached the eighth grade. And just 22 of Friday’s grads will be moving on to the new Success Academy High School of the Liberal Arts, which is set to open this fall, while 10 opted for other high schools.

None of the 32 grads, however, will be attending any of the city’s eight elite public high schools, even though Harlem Success Academy 1 ranked in the top 1% on state math tests this year and in the top 5% in reading — a fact Moskowitz herself proudly highlighted.

 “We are incredibly proud of our eighth-grade graduates . . . who are proving that zip code does not have to determine destiny,” Moskowitz said in a written statement.

A network spokeswoman confirmed Tuesday that 27 eighth-graders took the Specialized High Schools Admissions Test last fall, but none scored high enough to be offered a seat at one of the elite high schools that rely on the test, like Stuyvesant, Brooklyn Tech or Bronx Science.

Citywide, some 26,000 eighth-graders took the specialized high schools test in the fall of 2012, and 20% were offered a seat. So you’d expect a minimum of five or six students from Success 1 to score high enough to get into one of the elite schools.

...

Still, if Harlem Success students had matched even the 12% admission rate for black and Latino students who take the test, you’d expect at least three of the Moskowitz students to have been admitted.

“We were shocked that none of our students was offered a seat in a specialized high school,” one parent told the Daily News.