Tuesday, January 8, 2013

Metrics on education

Matt Yglesias has a post up about the StudentsFirst report claiming that outcomes are complicated to measure.  While this point is, in the abstract, true, it is informative to see what the highest ranked state (Louisiana) does in the grade 8 reading tests results that he shows.  He gives a number of groups: Everyone, Africian American, Latino, Low income, and White.  Louisiana is in the below average group for all groups except Latinos (where they are average).  It is also worth noting that Latinos make up about 3% of Louisiana's population, whereas African Americans make us around 34% of the population.  So they get average results only in a very tiny minority population.

Now there are lot's of reasons why a school system might be doing the best that they can and student results are divorced from a lot of complex social phenomenon.  However, when the top ranked states is below average on most student outcomes and above average for no populations that should be concerning. 

Now maybe the reforms have been too recent to have results.  But if reforms have very long lag times then we have another problem -- how do we properly evaluate the quality of reforms if it takes a decade to be seen in the test results? 

UPDATE: See here for the actual correlation coefficients.  Consider:
Looking more rigorously at the results, the correlation coefficient on the rankings in the StudentsFirst report card with state rankings on reading scores is -0.20. (The correlation coefficient is a measure of the similarity of two sets of numbers, ranging from -1.0, completely dissimilar, to +1.0, perfect similarity.) That’s not a large number, but the negative sign means that the correlation is in the wrong direction: the higher the StudentsFirst score, the lower the NAEP reading score. The correlation on math is even worse, -0.25.

It's not a good sign when the outcomes data is in the wrong direction. 

EDIT 2: Missing link inserted


  1. As a matter of basic social science, what should concern one is not the absolute level of a state's performance now but the counterfactual (what would its performance otherwise be).

  2. This calls to mind the classic example of worrying that police might cause crime because police presence is correlated with criminals. Perhaps states with worse educational outcomes currently are the ones most likely to try new reforms. The question will be whether those reforms pay off in the future.

  3. Stuart, I agree with the issue of the counterfactual. That being said, what is needed now is a timeframe for improvements to happen. Michelle Rhee started her DC reforms in 2007. What is the correct time window for DC to no longer be dead last in the rankings?

    The real surprise was Texas, which got an overall rating of D despite being an early adopter of education reform and, once you account for minority students, doing quite well on outcomes for a large state.

    Anomolies like that are what make me question the overall rankings.