Implications of the DC primary on education

Mark and I have been talking a lot about education. Today Seyward Darby has an article on the implications of Michelle Rhee leaving. What struck me as most interesting is that there are three concerns raised:

1) The fate of the new teacher evaluation system

2) That performance pay, based on IMPACT score, might not go forward

3) There could be increased regulation of character schools

Mark has commented on issues with a focus on narrow methods of teaching evaluation. But I think it is safe to say that items 1 and 2 depend critically on the quality of the method used.

As for number 3, deregulated private educational entities have geenrated concerns as the quality of instruction is hard to measure.

So everything hinges on how good the IMPACT evaluation score is. Now, this score is a hybrid (as opposed to pure standardized test scores) so it might not be awful:

It rates its teachers based partly on how well they improve student learning from year to year, and partly on intensive classroom observation by their supervisors.

However, it does not address the concern about teachers learning to game the system. Many ranking systems work very well at first and then people learn to optimize against them. Just consider how many articles there are on how to improve your FICO score.

But the core ideas remain to find a way to measure teacher performance on some sort of low dimensional scale and to reduce regulation of private sector competitors. It is unclear that these approaches will get at the key issues, even if (due to an extraordinarily dysfunctional school system) they may produce some improvements in the Washington DC schools.

More on the SRI KIPP report

This is a fairly small case of researchers being (deliberately?) obtuse, but it's representative enough to merit a brief post.

First look at this passage from Wikipedia:

Some observers, such as the authors of The Charter School Dust-Up,^[5] say that KIPP's admission process self-screens for students who are both motivated and compliant, from similarly motivated and compliant—and supportive—families. Parents must commit to a required level of involvement, which rules out badly dysfunctional families.

You'll notice this is consistent with some points Joseph and I have been arguing for awhile about how selection and peer effects can affect a school.

Now pay close attention to the following from SRI:

KIPP schools’ higher-than-expected test score results draw both attention and claims that they “cherry-pick” high-achieving students from poor neighborhoods. This is the first report to closely scrutinize the praise and criticisms associated with KIPP, as well as key challenges facing Bay Area KIPP schools today.

In the three KIPP schools where they were able to draw comparisons, SRI researchers found that students with lower prior achievement on the CST were more likely to choose KIPP than higher-performing students from the same neighborhood, suggesting that, at least at these schools, cherry-picking does not occur.

Did you catch the shift? Like most areas, academic success is largely determined by attitude and work habit. This is particularly true in a school with extended hours of instruction and a longer year. There is every reason to believe that these traits are strongly selected for here, just as suggested by the Economic Policy Institute.

The Charter School Dust-Up came out from the EPI in 2005, three years before the SRI report, but the SRI researchers never directly acknowledge the EPI's conflicting findings. Instead they pull the researchers' version of the old school kid's trick of pretending to mishear an inconvenient question.

Calling Yves Smith

(Via Andrew Gelman) John Sides has good take-down of Steven Hayward's odd screed for the American Enterprise Institute's house journal. Here's a choice exchange.

From Hayward:

Of course, this is just the silly stuff. The real problem with academic political science is its insistence on attempting to emulate the empiricism of economics and other social sciences, such that the multiple regression analysis is considered about the only legitimate tool of the trade. Some regressions surely illuminate, or more often confound, a popular perception of the political world, and it is these findings Klein rightly points out. But, on the other hand, I have often taken a random article from the American Political Science Review, which resembles a mathematical journal on most of its pages, and asked students if they can envision this method providing the mathematical formula that will deliver peace in the Middle East. Even the dullest students usually grasp the point without difficulty.

And from Sides:

This is sort of bizarre. Let’s leave aside the notion that “multiple regression analysis” is the “only legitimate tool.” That’s the impression of someone who doesn’t read much political science. I’m more interested in Middle East peace. Here’s my question: if Hayward picks up the American Economic Review, does he envision that their mathematical formulas will produce global prosperity? That’s the standard to which he seems to hold academic research. If so, he should be disappointed by virtually the entire corpus of social science, and perhaps by a decent bit of the hard sciences as well. After all, there’s still that cancer thing.

I'm going to have to take Sides' side here (and not just for the wordplay), but what really struck me as amusing was the use (starting with the subtitle) of economics as the gold standard of empiricism. Is that really the way you want to play it give the current state of economics in general and macroeconomics in particular?

This is so funny it made me cry

From Dr. Girlfriend.

Is this not every bio-statistician’s worst nightmare?

They just don't care

I am sure that most people in the news media watch every episode of the Daily Show and the Colbert Report. Then they get up the next morning, go to work, and do their God-damnedest to drag American journalism to its lowest point since Benjamin Franklin Bache accused George Washington of collaborating with the British.

They know what they're doing. They just don't care.

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
Islamophobiapalooza
www.thedailyshow.com
Daily Show Full Episodes Political Humor Tea Party

So which one is it?

DeanDad brings this point up in a post today. It's not related to the main point of the post but it is certainly worth noting:

Hell, while we’re at it, let’s make a point of generating enough math teachers so that every state in the country can require four years of math in high school. Get the public K-12 system up to basic competence, and see what happens.

I think, at some level, we need to decide what the priorities in education are. Is there a shortage of competent math teachers? If so then a process in which we make it harder to become a teacher would seem to be counter-productive. After all, we are also grappling with real issues of potential reductions in the number of teachers due to budgetary constraints.

I think that failing to decide on these issues are at the heart of my concerns about education reform. Reformers point to schools like KIPP that have attrtion rates that make them infeasible as a national model. So that doesn't seem to be a way forward. But in the actual education system, we are discussing reducing resources and trying to compensate with higher quality.

Now, add in that the metrics used to evaluate schools have serious concerns (as pointed out by a wide variety of researchers) and it gets hard to see what the road forward looks like. Clearly, if we are reducing resources to education then we can't achieve this with a higher investment. We might gain some efficiency by breaking contracts with current teachers (over tenure and pensions) but such actions tend to increase costs in the long run.

So I think that the real thing that I want to see out of educational reform is specific proposals. Honestly, I suspect that a series of initiatives at the school district level (focusing on the issues in each area) might be the way to go. But I worry that the current approach seems far too focused on test scores and not on the actual process of education.

The appeal of bad ideas -- why do rational executives keep trying ideas that never work?

A while back my friend Brian (of Ultrasonic Remote) and I found ourselves on the subject of TV remakes, shows that kept the title and general premise of an old hit but rebooted it with a new cast. After a while a pattern became obvious, the few successes all had something in common: they were all science fiction or fantasy shows (and even there the odds were not good).

When we went out of that subgenre, we couldn't come up with a single success. Many of the shows were high profile affairs with talented casts (Kojak, Dragnet, the Fugitive) but we are still talking about a failure rate of pretty much 100%.

Eventually, some show will break the streak. Perhaps it will be Hawaii Five-O or the new Rockford Files (after all, how hard could it be to replace James Garner?), but this still begs the question, why do highly paid executives in a tremendously competitive field insist on trying things that have repeatedly and consistently failed?

(Acknowledging) Failure Is Not an Option

Here at OE, we have been spending a lot of time discussing the horrible student attrition rates in Bay area KIPP schools, but when I first came across the statistic, I hesitated before using it in a post. The number was so absolutely damning that I had to wonder about the source. Had some anti-charter school group gone through all the data and cherry-picked the worst thing they could find?

So I looked up the source of the statistic and found a veritable sales brochure for KIPP. The article opened with the headline:

"New Study Finds San Francisco Bay Area KIPP Students Outperform Peers"

followed by the subtitle:

"Combination of Key Features Contributes to Success, Provides Lessons for Other Public Schools"

The rest of the article continues in the same vein before finally getting to this:

As researchers analyzed the student achievement data and KIPP’s approach, they also identified challenges facing Bay Area KIPP schools, including high student attrition rates, teacher turnover, and low state and local funding. For example, 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade. Annual teacher turnover rates have ranged from 18 to 49 percent since 2003-04.

Putting aside the human cost paid by the students who fell into that sixty percent and their families (which is huge), the selection/attrition process described in this post results in a school filled with roughly the top quartile of hard-working, dedicated students. All but the most incompetent administrators will have spectacular results under those circumstances. The performance described here ("In most grades, Bay Area KIPP students make above-average progress compared with national norms, and four out of five KIPP schools outperform their host district.") is, if anything, on the low end of what we would expect yet it is written up in the most glowing terms imaginable.

When it comes to charter schools, there's a long tradition of bad news stories with positive or at least neutral headlines. Consider this article from the New York Times. The key paragraph was:

But for all their support and cultural cachet, the majority of the 5,000 or so charter schools nationwide appear to be no better, and in many cases worse, than local public schools when measured by achievement on standardized tests, according to experts citing years of research. Last year one of the most comprehensive studies, by researchers from Stanford University, found that fewer than one-fifth of charter schools nationally offered a better education than comparable local schools, almost half offered an equivalent education and more than a third, 37 percent, were “significantly worse.”

And what was the headline for this grim finding?

"Despite Push, Success at Charter Schools Is Mixed"

I genuinely believe that there are some wonderful charter schools out there and we can greatly improve education by building on these successes, but in order to do that we need to recognize and honestly admit the failures.

Up until now, that honesty has been in short supply.

Substituting metrics for judgement

This episode of This American Life concludes with a fascinating account of the manipulation of crime statistics. Drop by, take a listen, give them a buck.

If reformers are looking for an indictment of our educational system...

I'd like to put some finer shadings on Joseph's recent post. I think I speak for most former and current teachers when I say that my favorite part of teaching was teaching -- explaining concepts, helping students working through problems, convincing them that they can not only do math, they can actually be good at it.

Helping a student who has shown an interest in learning always takes a high priority, but at some point something has to give. When I was teaching I routinely worked fifty plus hours a week when you count grading and making lesson plans. Add being on call ten or more hours a week and you will eventually have to make a choice. You can blow off the kids, you can blow off the grading and lesson plans or you can find out how long you can go without seeing your family or getting over six hours sleep.

As far as I can tell, the underlying assumption of the KIPP business model is an unlimited supply of Erin Gruwells. In case you've forgotten, Ms. Gruwell was the... let's say, inspiration for the character played by Hillary Swank in the movie Freedom Writers. Ms. Gruwell has managed an exceptional career but as a teacher, she followed a very common trajectory. She came into the field with high expectations and tremendous energy, worked incredibly long hours at a fantastic pace, achieved some early successes then bailed in less than five years.

KIPP burns through these teachers at an extraordinary rate ("Annual teacher turnover rates have ranged from 18 to 49 percent since 2003-04"). This is possible in large part because there are less than twenty-seven thousand students in the KIPP system. By charter school standards, this is substantial but there are over sixty-million K through 12 children in the U.S.

A number of people think KIPP is a scalable model for the nation. Many of these people hold advanced degrees from some of America's most prestigious universities. If reformers are looking for an indictment of our educational system, I can't think of a more damning one.

Cost of Education

I was speaking with Mark about this post and an interesting point came up. Teachers already have long and structured hours (pre and post class supervision, class hours and lunch duty) plus work at home (marking -- which at the high school level is never finished during the workday). KIPP has teachers also available by cell phone. In the private sector it generally involves a pretty hefty pay increase to put an associate on a pager.

When we talk about the need to increase our investment in education, are we really prepared to increase either the number of teachers or the median salary of teachers? After all, we are currently talking about teacher layoffs. Perhaps Mark, who has more practical experience in this matter, can comment?

Reflections on the KIPP posts

I want to explicitly emphasize this point in Mark's last post:

One is the eighty/twenty rule. Some students take more time than others and, not surprisingly, the students who lower a school's test average and management metrics are the ones who consume the most time and resources.

In the light of the KIPP website (which he discusses here):

Having teachers available by cell phone after school for homework help

Now, let us consider two effects of this policy. One, the school day is a lot longer for a teacher than it used to be. Add in the rigorous extracurricular activities and the (seemingly 24 hour on call status) and at best you are creating an environment where investing more resources in teaching creates better outcomes.

But I don't think that this is going to be the main effect. If you consider Mark's eighty/twenty rule, you create enormous incentives for teachers to gently nudge out the student who calls for an hour and half to discuss homework every single evening. Even if the teachers at KIPP don't do this (a proposition of which I would be highly sceptical), I would be deeply concerned about what whether this approach was scalable.

It think it would be critical to determine if an approach was scalable before basing massive reforms off of it. I am happy for innovation and improvement in education to continue but I think that we should proceed carefully and based on the evidence. In education, for reasons that evade me, people seem to be very non-critical about the evidence.

Make that most children left behind

In an email, Joseph accused me of burying the lede when I discussed KIPP charter schools in an earlier post. The paragraph in question was this quote from Wikipedia:

"In addition, some KIPP schools show high attrition, especially for those students entering the schools with the lowest test scores. A 2008 study by SRI International found that although KIPP fifth-grade students who enter with below-average scores significantly outperform peers in public schools by the end of year one, "... 60 percent of students who entered fifth grade at four Bay Area KIPP schools in 2003-04 left before completing eighth grade."[7] The report also discusses student mobility due to changing economic situations for student's families, but does not directly link this factor into student attrition. Six of California's nine KIPP schools, researched in 2007, showed similar attrition patterns.[citation needed] Figures for schools in other states are not always as readily available."

Administrators have long known that the simplest and most reliable way to improve a school's performance is by selection and attrition of the student body. This works in the obvious, direct ways -- if you drop the kids who disrupt class and/or can't master the material, test scores and classroom management metrics will go up -- but there are at least a couple of indirect effects that are as, or more, powerful.

One is the eighty/twenty rule. Some students take more time than others and, not surprisingly, the students who lower a school's test average and management metrics are the ones who consume the most time and resources.

Even more significant are peer effects. K through 12 students are particularly sensitive to perceived social norms. By selectively removing certain students from the population, you can easily create a high degree of conformity to an almost ideal set of behaviors and attitudes.

Most educators look at getting rid of students as a last resort. The prevailing attitude is that you are there to help all the kids, not just the easy ones. You will find exceptions, of course, like principals who are a little too eager to expel certain students or find ways to influence which students are assigned to other schools, but they are forced to work around rules that discourage this behavior.

For charter schools, though, these selective factors are built into the system. Here's the deal that college prep charter schools offer, if the students go through an involved application/induction process, take more difficult courses during a longer school day and do more homework, they will have a better chance at academic and professional success. These schools have automatically limited their pool of applicants to kids who consider academic and professional success both desirable and attainable and who come from supportive, involved families that are willing to make a real effort to give their children a chance to succeed.

(quick clarification: we are talking about selection processes that favor certain attitudes and behaviors. We are NOT talking about favoring students with high test scores. Many reporters and, God help us, researchers have failed to grasp this distinction.)

Charter schools segregate out a group of students who tend to be, to put it bluntly, easy to teach. This is not an entirely bad thing. Though there are concerns about this being a zero-sum-game, there is something to be said for making sure that every neighborhood has at least one educational bright spot.

It is, however, an entirely bad thing when dishonest or naive observers evaluate these schools without taking these systemic advantages into account and it is worse still when unscrupulous administrators try to build on those systemic advantages by cooking the data through selective attrition.

If you start with a student body made up almost entirely of kids who want to be in school, avoid gangs and graduate from college, who are supported by families with the same goals, you should expect relatively low attrition. When you see the opposite, you should certainly be suspicious. Even the most inept administrator can look good if you allow him or her to pick the most promising students out of an already select group.

The 'I' stands for 'Ironic'

Check out bullet two. (From the KIPP website)

At KIPP schools:

• Parents are encouraged to be involved and to contribute;
• Teachers have the freedom to innovate and never give up on a child;
• Students work hard and come to school ready to learn.

Update: While you're here, why not take a look at the next post

Rationality

From Econned page 148:

Greenspan was hardly alone in his dogmatic belief in the wisdom of leaving “free markets” to their own devices . . . Judge Frank Easterbrook and Daniel Fishel, then University of Chicago Law School dean, asserted in 1991: “[A] law against fraud is not an essential or necessarily important ingredient of securities markets.”

I find this line of thinking to be rather remarkable. The two classic libertarian roles for government are to prevent force and fraud. Markets can’t be efficient if information is missing or removed – in a world where identities can be re-invented then you simply can’t have a national (let alone) global market if fraud is a permissible business strategy. The only way to trust people with your money or in business deals would be a long history of relationships.

This line of thinking requires one to think that there are no information asymmetries in the real world. Furthermore, a rational agent permitted to engage in fraud might be willing to destroy their relationship for a large enough pay-off. Yves Smith gives an example of how this thinking is leads to odd conclusions:

Easterbrook went so far as blocking a plaintiff from presenting a case that argued that an auditor has assisted in a fraud. The judge claimed it would be “irrational” for the accountant to behave in that way, given his interest in preserving his reputation.

Of course, one could argue that all potential criminals have an interest in preserving their reputation. But one might very well risk their reputation for a sufficiently high pay-off – especially if one is highly unlikely to be prosecuted.

I do not think that this type of thinking is universal among “free market” advocates but it is extremely concerning that these arguments can be advanced without public mockery.