Comments, observations and thoughts from two bloggers on applied statistics, higher education and epidemiology. Joseph is an associate professor. Mark is a professional statistician and former math teacher.
I find it fascinating how the rules change between disciplines. In Epidemiology, the idea of a supervisor being an author on papers resulting from a doctoral discipline is completely non-controversial (and standard practice).
I think it is the difference in resources. The question is asked:
If, as an undergraduate, you handed in a paper that had partially been written by someone else, you would be sent down to the Dean's office, read the riot act with regard to academic integrity, and (probably) given an automatic F in the course.
Why doesn't the same thing happen with respect to a PhD thesis?
In Epidemiology the answer is that there would be no PhD theses under this rubric. Data takes too long to collect and analyze. Furthermore, granting agencies are simply unwilling to give massive amounts of grant funding to a PhD student. All of our research is deeply collaborative and single authored work can only happen in areas quite abstracted from reality or for very senior people who have access to data in a way that others do not.
When someone says something outrageous and stupid, there's a tendency (particularly on the blogosphere) to lock in on the outrageous part and give the stupidity a pass. Today's case in point comes from David Broder (via a suitably appalled Mark Thoma):
What else might affect the economy? The answer is obvious, but its implications are frightening. War and peace influence the economy.
Look back at FDR and the Great Depression. What finally resolved that economic crisis? World War II.
Here is where Obama is likely to prevail. With strong Republican support in Congress for challenging Iran's ambition to become a nuclear power, he can spend much of 2011 and 2012 orchestrating a showdown with the mullahs. This will help him politically because the opposition party will be urging him on. And as tensions rise and we accelerate preparations for war, the economy will improve.
I am not suggesting, of course, that the president incite a war to get reelected. But the nation will rally around Obama because Iran is the greatest threat to the world in the young century. If he can confront this threat and contain Iran's nuclear ambitions, he will have made the world safer and may be regarded as one of the most successful presidents in history.
Broder is, of course, suggesting just that, but while everyone is reacting to the indecency or, like Dean Baker, pointing out that infrastructure spending has the same effect, no one seems to be focusing on the stupidity of the analogy. So here I go in handy list form:
1. We don't have a draft. This war would mainly be fought with the forces at hand with minimal impact on unemployment;
2. WW II required a massive build-up in our air force and navy. How many more carriers are we going to need if we go war with Iran? In other words, this war would do little to absorb excess capacity;
3. We had a tightly controlled economy during the war that created a build-up in consumer demand. How likely is it that a GOP Congress would go along with that?
4. Immediately after the war, we extended a huge and unprecedented social safety net for the returning forces. Any chance of that happening?
If you're going to suggest fighting a war for economic gain, at least try to pick one that would actually do. Otherwise, you're being evil and stupid.
I'll be taking a break from p-values and peer effects tomorrow to watch the latest instalment of Sherlock now airing on Masterpiece Mystery. Mark Gatiss and Steven Moffat provide the scripts. That alone should be enough to get you to tune in.
Before I jump back into the weeds here, let's step away for a paragraph and remind ourselves that the question of the moment is "Do charter schools provide a good, workable model for the overall education system?" This is a high bar, but it's one that the reformers themselves set. There are things that many charter schools do which tend to improve student performance and attitude, but which would lose their effectiveness if applied to the general population.
Which brings us to the volunteer effect.
In Influence, Robert Cialdini observes that "we accept inner responsibility for a behavior when we think we have chosen to perform it in the absence of strong outside pressures." (I couldn't find my copy but it's on page 97 of the library's.)* It's part of the chapter on commitment which is filled facts and examples that are relevant to the current education debate.
Indoctrination plays a large part in the culture of schools like KIPP and HCZ. This isn't a bad practice -- all schools do it to some extent -- but it is particularly important when a school tries to greatly increase student work load. If you are assigning hours of homework every night and study sessions every Saturday and you expect reasonable compliance, you are going to have to make the students believe that hard work is the right thing to do and that it will lead to large rewards.
That kind of belief modification is easiest when subjects see themselves as volunteers who not only chose to engage in the accompanying behaviors but who actually made an effort to do so. This works well in the current incarnation of charter schools, but everyone can't be a volunteer. When educators show the ability to inspire these beliefs in the kids who don't want to be there, then we'll have something.
* As part of this discussion, Cialdini also mentions that small rewards tend to create more long-lasting behavior changes than large rewards. This sets up an interesting topic for a future blog post on economics and compensation.
I came across this while looking up some background for an upcoming post and clicked the link mostly of a sense of morbid curiosity (cynical people should generally avoid the business section of their neighborhood book store), but I was very pleasantly surprised.
Covers everything from Wealth of Nations to Blackhawk Down. Definitely worth a browse.
Messages are always a bit strange when you're not the target audience, but usually you can manage some level of understanding if you look at it from the target's perspective.
I don't agree with this protester's tee-shirt but I understand how he feels. On some level we all consider politicians in the other party liars and I suspect we have all indulged in the fantasy of saying some variation of "you lie" to a politician we particularly dislike and disagree with.
But sometimes, no matter how you try to see things from the other guy's vantage, you simply can't understand how something provokes its intended response.
Carly Fiorina's campaign spent tens of millions of dollars on an ad showing a clip of Barbara Boxer insisting that a witness at a Senate hearing refer to her as 'senator' saying "I worked so hard for that title." Now another group is using this phrase as a tagline.
Perhaps it would be different if I could have gotten Zucker's video to work, but I can't imagine how hearing a senator say "I worked so hard for that title" would generate much of an emotional response. Kay Bailey Hutchison worked hard to get into the Senate. So did Orin Hatch. So did every senator I can think of.
Is there an emotional resonance I'm missing here? Or is this a miscalculation on the part of Fiorina and company (keep in mind, these are the same people who thought a Bond villain would make a good spokesman).
A big part of being a statistician is knowing when to get nervous, knowing when cutting a small corner could produce massive bleeding. One of those cut corners we pay particularly close attention to is the open-label trial, where the subjects know exactly what treatments they are getting.
In a perfect world, neither the subjects in a trial or the people administering the treatments would ever know who was getting what. This double-blind approach protects us from the placebo effect, which has an unfortunate way of popping up whenever humans are the subjects of research.
At the risk of being obvious, it is next to impossible to perform a double-blind experiment in most areas of educational research -- everyone knows who got the treatment and who didn't -- but that doesn't mean that the underlying reasons for preferring double-blind tests aren't there. We routinely allow for the possibility that the placebo effect can affect pretty much everything from surgery to the immune system. Does ignoring the possibility that it might affect student performance seem like a safe assumption?
If anything, education is a textbook example of an area where we would prefer not to use an open-label approach. We are working with a test population that's highly suggestible and treatments that rely heavily on the subjects' attitude. Under these conditions, telling subjects that they are about to receive a treatment is highly likely to bias the results.
And in the case of charter schools, students aren't just told they are about to receive a treatment; they are often told, in the most dramatic way possible, that they are about to have their lives transformed. Pedagogically, this is a good idea. It helps establish the belief that the student will succeed, a belief that can easily become self-fulfilling. Statistically, though, it greatly muddies the waters.
Watch the first minute of this Sixty Minutes segment. Look at the expressions of the father leaping out of his seat with excitement and the mother and daughter crying with joy. You'd have to be emotionally dead not to empathize with this family's feelings, but you'd also have to be a poor statistician not to wonder if those emotions contributed to student success.
update: I probably should have mentioned that there is reason to suspect that group dynamics may amplify the placebo effect here.
This is the second part of a reply to a previous comment about counseling out disruptive students. In the first part I discussed how charter schools' greater freedom to get rid of problem students makes reliable public-to-charter school comparisons difficult, but as important as that question is, there's another that's even more fundamental when discussing these practices: what exactly do we want charter schools to do?
We have two basic choices:
The first is that charter are what all schools should eventually be. In this model, charters provide the template for the American education system. If this is what we're asking from them, then charters' problems dealing with disruptive and non-cooperative students is a serious failure;
If, however, we look at charter schools as niche programs designed to target specific areas and subpopulations, then counseling out student for academic or behavioral reasons may not be a problem at all. If the purpose of these schools is to allow room for experimentation, pump additional resources into under-served areas and provide a better match for certain kids who aren't getting what they should from the one-size-fits-all approach, then counseling out is a necessary part of the model.
Most champions of charter schools would probably pick the first model but most of the major criticisms that have been made recently about charters schools (data biasing issues, accusations of cherry-picking, questions about scalability) largely go away under the second model.
I have a feeling we'll be coming back to this one.
This comment from Michael Bish raises a couple of big questions:
One important point is that there is a big difference between counseling a student out because they have low test scores and counseling them out because they are actively disrupting the education of their peers.
The first issue here involves metrics. The idea that you should build a school system around a handful of test scores is one of the central tenets of the reform movement. Test scores are supposed to drive funding, contracts, allocation of resources, evaluations, bonuses, even terminations, but if you give certain schools more freedom to get rid of disruptive students the whole system breaks down.
Disruptive students not only take up time from the teacher that could be used in instruction; they also make it difficult for students to concentrate and tempt other others to act up. The result is that everyone's test scores go down.
By getting this student transferred you've not only raised the scores of an entire classroom of your students; you've lowered the scores of a comparable number of students in a public school in the same area. Since your school's evaluation, funding, and future contracts are dependent on your performance relative to other schools in the area, you can get a substantial double lift out of that single transfer.
And since public school to a large extent have to work with the students they are given, charter schools always win this one.
And the loneliest orphan of them all is over-the-air television. It's the kind of story you would expect to find in a Lemony Snicket novel, complete with evil guardian trying to kill it off to get to the family estate (bandwidth).
OTA television has few friends. Cable and broadband get all the attention while the media has been committed to the death of broadcasting/networks for more than three decades now.
Of course, one of the problems with being committed to a narrative is that it forces you to ignore the details that don't support the narrative and these have an unfortunate way of being the interesting ones.
Case in point: this story on the growth of Univision, though a bit credulous ("Univision set to become top U.S. broadcast network"), contains some impressive statistics about the growth of the network. It does not, however, contain any mention of this:
Univision is concerned because nearly 28% of Hispanic households — and 43% of homes where Spanish is the primary language — watch TV only via over-the-air transmissions, according to a 2005 National Association of Broadcasters report to the FCC.
Given that Univision skews toward the Spanish-language only households, that means a big chunk of its audience is coming in over the air. That means that selling off that part of the spectrum would have big consequences for the fast growing area of Spanish language media. That's an awfully important detail to leave out.
(This post also appears at Education and Statistics.)
Paul Tough writing in Slate recounts the following:
In Whatever It Takes, in one of the chapters on the Promise Academy middle school, I describe the impact of the KIPP schools in the Bronx and Harlem on the Promise Academy’s leaders and staff. This was during the first few years of the Harlem Children Zone’s middle school, which were a struggle, and those KIPP schools, which had very good test results, were for the Promise Academy administrators both a standard to be aspired to and a frustrating reminder that their own students weren’t performing at the same high level as KIPP’s students.
Terri Grey, the Promise Academy principal at the time, believed the attrition issue was part of what was holding her school back. As she put it to me in one conversation, “At most charter schools, if the school is not a good fit for their child, the school finds a way to counsel parents out”—to firmly suggest, in other words, that their child might be happier elsewhere. “Whereas Promise Academy is taking the most disengaged families and students and saying, ‘No, we want you, and we’re trying to keep you here, and we don’t want to counsel you out.” That policy made it impossible, she believed, for the Promise Academy to achieve KIPP-like results.
I’m not entirely convinced that that was the real problem at Promise Academy—or that the KIPP schools in New York were actually “counseling out” a significant number of students. But I do think it’s true that Geoffrey Canada’s guiding ethic has always been to go out of his way to attract and retain the most troubled parents and students. And that makes running a school, or any program, more difficult, even if it makes the mission purer and, in the end, more important.
For reasons I'll get to later, I suspect that the number of students you have to "counsel out" to have a significant effect on a school's test scores is lower than Mr. Tough realizes, but there are a couple of more important points.
The first is that selective attrition is recognized as a serious issue not just by critics of the reform movement but by responsible people within the charter school community.
The second is that all charter schools and charter school administrators are not interchangeable. There are some gifted educators with great ideas in that system. We've spent almost two decades overlooking the flaws in charter schools. It would be a serious mistake to try to compensate by overlooking the strengths.
This post by Megan McArdle was interesting for the context. I am certain that I will say something I regret sooner or later; it's probably a good thing not to be judged by hasty words.
But I also think it is important to remember that all quotes can be made to look bad out of context. In the "sound-bite" environment of the modern world it is already hard enough to discuss complex issues (*cough* education reform *cough*) without adding in misleading quotes.
I received an email from Dr. DeWayne Davis, the principal of Audubon Middle School in Los Angeles, which was sent to several public officials. Dr. Davis said that local charter schools were sending their low-performing students to his school in the middle of the year. He wrote:
"Since school began, we enrolled 159 new students (grades 7 and 8). Of the 159 new students, 147 of them are far below basic (FBB)!!! Of the 147 students who are FBB, 142 are from charter schools. It is ridiculous that they can pick and choose kids and pretend that they are raising scores when, in fact, they are purging nonperforming students at an alarming rate--that is how they are raising their scores, not by improving the performance of students. Such a large number of FBB students will handicap the growth that the Audubon staff initiated this year, and further, will negatively impact the school's overall scores as we continue to receive a recurring tide of low-performing students."
Ravitch concludes:
Doing better than an under-resourced neighborhood school is not the same as getting "amazing results." Very few charters do. Probably less than 5 percent. Charters are not a silver bullet. They are a lead bullet. Their target is American public education.
This is just par for the course for modern conservatism: have private systems skim the cream, and leave the public sector to clean up an impossible mess. When they can't, this supposedly shows the inability of government to solve problems.
I have a few points to add:
1. This is a brutal way to treat these kids. You build their hopes up, then crush them, then dump the kids in a new school in the middle of the year;
2. We are talking about getting an influx of students who are badly behind and who are ready to give up and/or act out. This will disrupt classes slightly less than having a nearby car alarm go off at random times once or twice an hour;
3. But I think Ravitch overstates the case against charter schools. I've dealt with some small, independent schools that have impressed the hell out of me and I can see them playing an important role in our system, though a radically different role than Arne Duncan sees.
This post from Worthwhile Canadian Initiative is probably the best reason for my recent lack of activity in the blogosphere. Coupled with a lot of travel, I am quite behind.
Hopefully a temporary condition.
In the meantime, Mark has been putting up some great material that is well worth reading, so at least radio silence hasn't struck yet.
No time to blog this afternoon but if there was I'd join in this discussion on polls and momentum. While you're there, take a minute to check out the comments.
