The education reform movement relies heavily on anecdotes of remarkable, odds-beating schools, but when you take a close look at those schools that had significantly superior performance, some if not most of the difference in scores could be explained by selection and peer effects.
This isn't to say these schools weren't benefiting their students. Regardless of the reason, these kids were better off. Nor is this to say that these schools weren't doing something right. I can tell you that many are well-run and highly innovative.
But even taking all of that into account, selection and peer effects are huge and can swamp almost any other factor you can think of. These effects are seldom if ever adequately accounted for (And before anyone says the word 'lottery,' please take a look at this). This makes it all but impossible to accurately measure the impact of these schools but people like Jonathan Chait continue to cite them without any caveats.
I came across a segment of This American Life that beautifully captured my feelings on the subject. Just play the clip below and every time you hear 'heroin,' substitute in 'selection and peer effects' (you can just ignore the parts about crystal meth and pancakes).
From Kumail Nanjiani:
So remember, selection and peer effects are doing the heavy lifting.
I also read article after article about research supposedly proving the effectiveness of one educational innovation or another, thinking all the while, “But did they control for the self-selection effect? Did they distinguish causation from correlation?”
ReplyDeleteThey almost never do. A case in point is the Clifford Adelman study “Answers in the Toolbox,” with which I had a run-in a few years back when I was teaching high school math in Arizona and my district was pressured to institute certain kinds of AP classes because that study had given rise to this federal grant program:
http://www2.ed.gov/programs/apincent/index.html
which had given rise to this Arizona grant program:
https://www.azed.gov/accountability/apip/.
I just looked up the Adelman study today and found this description of it at Kathleen Plato´s site
http://www.newhorizons.org/spneeds/inclusion/teaching/plato.htm :
AP is a Predictor of College Success
"In 1999, a U.S. Department of Education study by Clifford Adelman titled "Answers in the Tool Box" examined the factors which could be used to predict college success – the attainment of the Bachelor's degree. Adelman was not examining factors that supported college admission; his research was focused on college completion. The study concluded, "No matter how one divides the universe of students. . . .a high school curriculum of high academic intensity and quality is the factor that contributes to a student's likelihood of completing a college degree." Courses such as AP and IB outranked, grade point, class rank, SAT scores as contributing factor. Furthermore, rigorous high school courses were shown to mitigate the effects of socioeconomic status. Just one AP course exposes a student to college-level work. Even if the examination is not taken or passed, the challenge of the course and the emphasis on critical thinking, study skills and increased content knowledge prepares a student for college work."
Note that last part especially – the implication that it has somehow been proved that even students who do not take or pass the AP exam are more likely to complete college just by virtue of having enrolled in an AP class.
The Adelman study did not prove that; there was nothing in it that in any way broke out students who were enrolled in AP but didn´t take or pass the test. (It´s been a while since I read it, but I don´t remember that it even quantified how many such students there were.) There was nothing in it that controlled for the two sorts of selection I would be willing to bet were in play among the AP-enrolled students: (1) They were motivated and willing to work hard and (2) In all likelihood there were prerequisites for enrolling in the AP classes in the first place, which may have well included minimum scores in previous classes. If some disaster on day 2 of the AP class had prevented it from going forward, those students enrolled still would have completed college.
MY COMMENT IS OVERLONG SO I AM CONTINUING IT AS A COMMENT TO YOUR OCTOBER 1 POST.