Comments, observations and thoughts from two bloggers on applied statistics, higher education and epidemiology. Joseph is an associate professor. Mark is a professional statistician and former math teacher.
Monday, February 14, 2011
Great moments in metawork
"To seek out new data and new analytic techniques."
The attendees were all experienced modellers and data miners, some fairly high ranking with commensurate salaries. Everyone in that room had something else they needed to be doing and, except for the senior manager present, I doubt that anyone present saw any real value in the exercise. Still, word had come down from the top that every distinct subgroup in the company needed its own mission statement so there we were, boldly splitting that famous infinitive one more time.
On the bright side, at least this was one time we didn't have to have a pre-meeting.
"The Economics of Blogging and The Huffington Post"
That might have been a mistake on my part.
(thanks again to Felix Salmon)
Concerns with data driven reform
Consider the effect of No Child Left Behind. I've seen a noticeable decline in basic math skills of students of all levels in the last 5 years. Every year, I will discovered a new deficiency that was not seen from the previous years (we are talking about Calculus students not able to add fractions). Yet NCLB was assumed to be "working" since the scores were going up. It seems that K-12 was devoting too much time preparing the students for tests, at the cost of killing students' interest in math, trading quality instruction for test-taking skills. Is NCLB a factor in the study? Are socio-economic factors examined in the study?
or CC Physicist who stated:
I look at what Asst Prof wrote as an indication that a Dean, chair, and mentor didn't do a good job of getting across the history of assessment. Do you know what "Quality Improvement" program was developed a decade earlier, and what the results were of the outcomes assessment required from that round of reaffirmation of accreditation? Probably not, since we have pretty good communication at our CC but all the negative results from our plan were swept under the rug. The only indication we had that they weren't working was the silent phase-out of parts of that plan. Similarly, data that drove what we did a decade ago were not updated to see what has changed.
I think these two statements capture, very nicely, the main issue I have with the current round of educational reform. One, if you make meeting a specific metric (as a measure of on underlying goal) a high enough priority then people will focus on the metric and not the actual goal. After all, if you don’t then your name could be posted in LA Times although with your underperformance on the stated metric. So we’d better be sure that the metric that we are using is very robust in its relation to the underlying goal. In other words, that it is a very good representation of the curriculum that we want to see taught and measures the skills we want to see students acquire.
Two, trust in evidence based reform requires people to be able to believe the data. This is one area where medical research is leaps and bounds ahead of educational research. A series of small experiments are attempted (often randomized controlled trials) while the standard of care continues to be used in routine patient care. Only when the intervention shows evidence of effectiveness in the trial environment is it translated into routine care.
In education, such trials are rare indeed. Let us exclude natural experiments for the moment; if we care enough to change the education policy of a country and to violate employment contracts then it’s fair to hold ourselves to a high standard of evidence. After all, the lotteries (for example) are not a true experiment and it’s hard to be sure that the lottery itself is completely randomized.
The problem is that educational reforms look like “doing something”. But what happens if the reforms are either counterproductive or ineffective (and implanting an expensive reform that does nothing has a high opportunity cost). The people implementing the reforms are often gone in five to ten years but the teachers (at least now while they have job security) remain to clean up the wreckage afterwards.
I think that this links well to Mark's point about meta-work: it's hard to evaluate the contributions of meta-work so it may look like an administrator is doing a lot when actually they are just draining resources away from the core functions of teaching.
So when Dead Dad notes: “Apparently, a national study has found that colleges that have signed on to ATD have not seen statistically significant gains in any of the measures used to gauge success.” Why can’t we use this evidence to decide that the current set of educational reform ideas aren’t necessarily working well? Why do we take weak evidence of the decline of American education at face value and ignore strong evidence of repeated failure in the current reform fads?
Or is evidence only useful when it confirms our pre-conceptions?
Metawork
As he liked to explain it, metawork is not, in and of itself, a bad thing. A certain amount is necessary for a well-functioning organization. It's not unusual for new companies to fail because of an overly rich work to metawork mixture.
But, my friend went on, metawork is like a gas -- it expands to fill all available space, both because it's easy to create metawork projects and because those projects can often be stretched to whatever time is available to them (you can always schedule an extra meeting). Furthermore, once it has established a foothold, it has a way of becoming part of the corporate culture.
There are also other reasons why companies tend to grow more metawork heavy as they mature and expand:
Major metawork initiatives tend to be top down (no customer ever said, "I like this company's products but I have a feeling they aren't having enough team-building seminars."). From a career standpoint, it is always a good idea to give a high priority to projects that people above you consider important;
Metawork projects almost always sound good. They have impressive sounding goals like improving efficiency, raising morale, making the company more nimble and responsive, or moving to data-driven strategies (more on that one in future posts). They suggest big-picture, forward-thinking approaches that make fixing problems like billing glitches seem prosaic, perhaps even trivial;
Metawork tends to be safer than the other kind. Let's say a company launches two big and badly-conceived initiatives, a new product launch and a 'data-driven' reworking of the project management process. The product sells badly and the new process eats up man hours without making projects run faster or smoother. Both end up costing the company about the same amount of money, but the product's failure is public and difficult to ignore while the process's failure is internal and can be denied with some goalpost moving and willing suspension of disbelief (something that's easy to generate for a VP's pet project);
As mentioned before, metawork isn't all pre-meetings and mission statements. Some kinds of metawork are essential (payroll comes to mind). Other kinds can help a company improve its profitability and stability (like employee morale studies in a labor-intensive industry with high turnover). Employee can be resistant to some of these good, important initiatives, but it's worth keeping a couple of facts in mind:
There is a lot of bad metawork out there;
The employees who most resent doing metawork are often the employees who are doing the most of the other kind of work.
Sunday, February 13, 2011
One more simple game for your weekend
Dice A {2,2,4,4,9,9}
Dice B {1,1,6,6,8,8}
Dice C {3,3,5,5,7,7}
The game has three rounds. First you roll against Pierre, then you roll against Blaise, then Pierre and Blaise roll against each other. The winner of each round is the one who rolls the higher number. The overall winner is the player who wins the most rounds.
Which die should you choose?
[Here's the relevant link (try not to look at the status bar -- the address gives away too much). It's a fun, trivial oddity but it raises some interesting questions about how numbers we trust can do unexpected things. More on that later.]
Saturday, February 12, 2011
Weekend Gaming -- Hexagonal Chess
Case in point, Gliński's hexagonal chess.
Gliński's chess variant is hugely popular in Europe (more than 100,000 sets have been sold). You can get the rules at my Kruzno site, but you can probably figure most of them out for yourself. The only pieces that might give you trouble are the bishops and,to a lesser extent, the knights.
Bishops come in three colors, which points out an interesting topological feature of a hexagonal grid which I'm betting you can spot for yourself.
It's a strange and intriguing game and yet another reason why every house should have a hexboard.
Friday, February 11, 2011
"Why the Efficient Market Hypothesis (Weak Version) Says Nothing about the Ability to Identify Bubbles"
Let’s put aside the possibility that even the weak EMH can be wrong from time to time. We don’t need to go there; the error is more basic than this.
Let’s put ourselves back in 2005. It is two years before the unraveling of the financial markets, but I don’t know this; all I know is what I can see in front of me, publicly available 2005 data. I can look at this and see that there is a housing bubble, that prices are rising far beyond historical experience or relative to rents. The “soft” warning signs are all around me, like the explosion of cheap credit, the popularity of credit terms predicated on ever-rising prices, and the talk of a new era in real estate. Based on my perceptions, I anticipate a collapse in this market. What can I do?
If I am an investor, I can short housing in some fashion. My problem is that I have no idea how long the bubble will go on, and if I take this position too soon I could lose a bundle. In fact, anyone who went short in 2005 and passed on the following two years are price frothery grossly underperformed relative to the market as a whole. Indeed, you might not have the liquidity to hold your position for two long years and could end up losing everything. Of course, it is also possible that the bubble could have burst a year or two early and your bets could have paid off. What the EMH tells us is that, as an investor, not even your prescient analysis of the fundamentals of the housing market would enable you to outperform more myopic investors or even a trading algorithm based on a random number generator.
The logical error lies in confusing the purposes of an investor with those of a policy analyst. Suppose I work for the Fed, and my goal is not to amass a personal stash but to formulate economic policies that will promote prosperity for the country as a whole. In that case, it doesn’t much matter whether the bubble bursts in 2006, 2007 or 2010. In fact, the longer the bubble goes on, the more damage will result from its deflation. At the policy level, the relevant question is whether trained analysts, assembling data and drawing on centuries of experience in financial manias, can outperform, say, tarot cards in identifying bubbles. The EMH does not defend tarot.
To profit from one’s knowledge of a market condition one needs to be able to outperform the mass of investors in predicting market turns, which the EMH says you can’t do. Good policy may have almost nothing to do with the timing of market turns, however.
The Good Principal Principle
Yet each school year thousands of principals beat the odds and do excel, women and men who love their leadership positions, relish the challenges and take pride in running schools that perform well year after year. Who are these people? And what are they doing that so many others aren’t?
“We know that principals matter for a school’s success, but we don’t know much about why and how they matter,” says Jason Grissom, an assistant professor of public affairs in MU’s Truman School of Public Affairs. Grissom and Susanna Loeb, a professor of education at Stanford University, are working to provide answers, thanks in part to a $1 million grant from the Institute of Education Sciences, the self-described “research arm” of the U.S. Department of Education.
“Our goal at the end of this study is to be able to offer some tangible recommendations for making principals more effective in terms of improving student outcomes,” Grissom says. “We’re excited about the proposal because it’s pretty ambitious. The kind of study we’ve proposed is essentially the first of its kind.”
At least we've gotten more efficient at some things
I Am Number Four is a young adult science fiction novel by Pittacus Lore, the pen name of authors James Frey and Jobie Hughes. The book was published by HarperCollins on August 3, 2010,[1] and has currently spent 6 weeks on the children's chapter of The New York Times Best Seller list.[2]
DreamWorks Pictures bought the rights to the film in June 2009; it will be released on February 18, 2011. The novel is the first of a proposed six-book series.[3]
Back in the old days, you used to have to publish one before they started the film series. By way of comparison, it took James Bond nine years, ten best sellers and the implied endorsement of JFK before it made it to the screen.
E-verify
And the results have been devastating. U.S. workers excited to start a new job are instead thrust into bureaucratic limbo as they try to sort out government mistakes. Their new employers hire, then fire them and never tell them why; or worse, they might never be hired in the first place and not know why . . . According to government reports, the program (even after years of work) has a stubbornly high error rate and well-documented problems in attempts to resolve those errors. According to the most conservative numbers, at least 80,000 American workers lost a new job last year because of a mistake in the system. If E-Verify were mandatory, that number would rise to 770,000
and
Ultimately the most brutal irony is that E-Verify doesn’t work. According to government-required audits, 54 percent of those not allowed to work in the U.S. were actually approved by the system.
Even worse:
The government mechanism to fix errors is a Kafkaesque tragedy. There is currently no court remedy to force Immigration and Customs Enforcement to fix an error. Many times those errors are as simple as an incorrect data entry or a name change, but in order to uncover the error, workers have to file letters with different parts of the agency seeking copies of their records
The part that is the most painful about this process is that being denied employment is a cost (especially in the current employment environment). What happens if we refuse a person employment (incorrectly) and their unemployment insurance is about to expire? Worse, if they do not know why they were flagged then it might be years before they find out what went wrong.
This doesn't seem like a good idea at all.
Administration
I’ll answer the question with another question. Good, strong, solid, peer-reviewed scientific data has made it abundantly clear that poor eating habits lead to obesity and all manner of negative health outcomes. There’s no serious dispute that obesity is a major public health issue in the US. And yet people still overeat. Despite reams of publicity and even Presidential support for good eating and exercise habits, obesity continues to increase. Why?
In other words, reform is hard to do even when you know where you want to go. In cases where the evidence is weak or where budgets are falling, the problem gets even worse. And, of course, right now we are experiencing a fall in most education budgets in the United States.
However, it was interesting how Dead Dad was unable to resist worrying about tenure as a barrier to reform:
There’s also a fundamental issue of control. Faculties as a group are intensely protective of their absolute control of the classroom. Many hold on to the premodern notion of teaching as a craft, to be practiced and judged solely by members the guild. As with the sabermetric revolution in baseball, old habits die hard, even when the evidence against them is clear and compelling. There’s a real fear among many faculty that moving from “because I say so” to “what the numbers say” will reduce their authority, and in a certain sense, that’s true. In my estimation, this is at the root of much of the resentment against outcomes assessment.
Even where there’s a will, sometimes there just isn’t the time. It’s one thing to reinvent your teaching when you have one class or even two; it’s quite another with five. And when so many of your professors divide their time among different employers, even getting folks into the same room for workshops is a logistical challenge.
Of course, accountability matters. Longtime readers know my position on the tenure system, so I won’t beat that horse again, but it’s an uphill battle to sell disruptive change when people have the option of saying ‘no’ without consequence. The enemy isn’t really direct opposition; it’s foot-dragging.
I think that this line of thinking may also be part of why there is such a huge push for reform of teacher job security. Administrators are under enormous pressure to reform the education system and teachers may be very resistant.
Of course, one element that may be left out is that the teachers may be resistant to change for good reasons. When you have been in an organization long enough, you realize that a lot of reform can be about trying "old ideas all over again". These reforms can be both time consuming and ineffective. They may even lower outcomes due to the friction of implementation.
Let us consider a business analogy. One way that corporations tried to handle bad outcomes is a series of "re-orgs". These changes in structure have two good properties. One, the people in charge seems to be doing something to address issues by making changes. Two, a series of re-organizations can make it very hard to track a long term pattern of bad management as units break apart too often for performance to be easily tracked.
The ability of tenured people to resist cosmetic reforms is, obviously very frustrating to administrators who have little ability to influence the organization but seemingly unlimited accountability. However, endless re-organizations did not, in the end, help corporations like General Motors. Instead, they may well have accelerated the decline by focusing on making changes that were more cosmetic than effective. So do we really need to import the worst practices of modern corporations into the educational system?
[This post is also relevant -- Mark]
The Principal Problem
This is particularly strange because, if you're looking for something to explain why a certain school is under-performing, you would obviously start by looking for a common factor, something that could explain why so many classes are bad and so many students are doing poorly. When you take out demographics and social factors, the only candidate left is administration, the people who hire and manage the teachers, who maintain overall campus discipline, who are responsible for how the school runs. Running a school is a tough job, but there are lots of great schools out there, both public and charter, so obviously it is possible to do it well.
I've always felt that every firing represents a failure of the hiring or management process, but if you have your heart set on reform by winnowing, it seems clear that administration should be the first to go. Unfortunately, administrators tend to be survivors (Ever been to a school board meeting? It's hard to avoid the cockfight analogy). Case in point...
U.S. Plan to Replace Principals Hits Snag: Who Will Step In?When I mentioned this story to Joseph, he drew a parallel between this and the financial crisis. In both cases we were told that we couldn't get rid of the people who screwed up because we supposedly needed them to fix the problem. It was hard to swallow with AIG; it's even less credible here. But it's what you expect from survivors.
By SAM DILLON
COLUMBUS, Ohio — The aggressive $4 billion program begun by the Obama administration in 2009 to radically transform the country’s worst schools included, as its centerpiece, a plan to install new principals to overhaul most of the failing schools.
That policy decision, though, ran into a difficult reality: there simply were not enough qualified principals-in-waiting to take over. Many school superintendents also complained that replacing principals could throw their schools into even more turmoil, hindering nascent turnaround efforts.
As a result, the Department of Education softened the hit-the-road plans for principals of underperforming schools laid out in the program rules. It issued guidelines allowing principals hired as part of local improvement efforts within the last two years to stay on, then interpreted that grandfather clause to mean three years.
Although the program created an expectation that most schools would get new leadership, new data from eight large states show that many principals’ offices in failing schools still bear the same nameplates. About 44 percent of schools receiving federal turnaround money in these states still have the same principals who were leading them last year.
Being politically skilled is valuable to an administrator, as is being media-savvy. This is not a bad thing. These talents can help administrators serve their students and promote their vision (look at Geoffrey Canada), but, as an old superintendent told me when I first started teaching (in somewhat more blunt language), you have to be aware of these talents and be careful when dealing with people like him.
Not surprisingly, superintendents have done pretty well for themselves in the reform movement. They have brought in additional money. They have managed to shift most of the attention from the damage done by bad administrators to the damage done by bad teachers. When colleagues actually were blamed for failing schools, they have frequently managed to shield them from any real consequences.
The past couple of years have even seen the emergence of the superstar school administrator. With the rise of Michelle Rhee and Joel Klein, what was a good gig now has at least the possibility of significant fame and fortune.
Having said all of this, it's important to step back and remind ourselves of some basic truths:
Whether you're talking about administrators or teachers or researchers or reformers, virtually everyone involved with education is there out of a deep concern for the education and general welfare of children;
At the same time, all of these groups will also tend to look out for their own self-interests. There is no contradiction here. We expect the police to have the interests of the public and of the police department in mind. We expect the same of firemen, journalists, the military and many others.
Administrators are very good at this game. There's nothing wrong with that, as long the people covering the game know the score.
Thursday, February 10, 2011
I originally had a mean headline but my better angels swooped in*
Admittedly, I have only skimmed the book, but it is not hard to find evidence that Dr. Ravitch has not left all of her highly conservative views behind. She blames the familiar bogeymen of the religious right for many of the problems in American public education, notably constructivism and whole language with the selective citing of easily refuted research. Her naive understanding of learning theory or learner-centered pedagogy is like that of a teacher education student or mom who just returned home from a “Tea Party” rally.In the education debate, you don't have a left/right divide; you have a Möbius strip.
*R.I.P. Gerry Rafferty
Do good teachers make difficult employees?
With few exceptions, experienced teachers are comfortable addressing large groups and with very few exceptions, effective teachers are comfortable demanding the full attention of those group. Along with knowledge of the subject, strong communication skills, and commitment to the students, a "when I talk, you listen" attitude is a defining trait of an effective instructor.
That doesn't automatically translate to a room full of kids sitting quietly while the teacher drones on. Often the result is just the opposite. Teachers are more likely to have looser classes with more student participation if they feel in control. As a rule of thumb, you should never be more than ninety seconds away from having every student seated and reading quietly. For really good teachers, even the most adventurous lesson plans fall into that ninety second radius.
Put another way, it comes down to authority. A teacher's job is to teach, counsel and objectively evaluate his or her students. A sense of authority is an essential trait for all these tasks but it's an incredibly annoying one to find in a direct report.
Good principals (and I've met some excellent ones) are masters at the difficult art of managing managers. They can exercise their authority in a way that actually enhances the authority of those under them.
Even with the best administrators, however, there is always an element of tension and it only gets worse with less competent principals and superintendents. This is something to keep in mind when you hear about plans to improve education by giving principals more authority to get rid of bad teachers. Sometimes bad doesn't mean incompetent; it means inconvenient. (I don't have the book in front of me, but Diane Ravitch' Death and Life has some notable examples.)