Comments, observations and thoughts from two bloggers on applied statistics, higher education and epidemiology. Joseph is an associate professor. Mark is a professional statistician and former math teacher.
I've got a new post up at the teaching blog about the differences in live presentation and other educational media. Check it out if that sort of thing sounds interesting, but if you do, you might want to watch Flight of the Phoenix first.
For years now, there have been two basic narratives when it came autonomous cars. The first is what I've called the ddulite version: driverless cars are just around the corner and they are about to change our lives in strange and wonderful ways if we can just keep the regulators out of the way. The second version is more skeptical: while lower levels of autonomy are coming online every day, the truly driverless car still faces daunting technological challenges and, even if those are met, these cars may not not have the often-promised impact. (You can probably guess which side I took.)
If you follow this story through the New York Times or the Economist, you are overwhelmingly likely to get the first version, You may not even know that this bright future is contested. If, on the other hand, you talk to the engineers in the field (and I've talked to or exchanged emails with quite a few recently), you are far more likely to get the second.
This recent Slate article by Lee Gomes is one of the very few to take the second approach.
For starters, the Google car was able to do so much more than its predecessors in large part because the company had the resources to do something no other robotic car research project ever could: develop an ingenious but extremely expensive mapping system. These maps contain the exact three-dimensional location of streetlights, stop signs, crosswalks, lane markings, and every other crucial aspect of a roadway.
That might not seem like such a tough job for the company that gave us Google Earth and Google Maps. But the maps necessary for the Google car are an order of magnitude more complicated. In fact, when I first wrote about the car for MIT Technology Review, Google admitted to me that the process it currently uses to make the maps are too inefficient to work in the country as a whole.
To create them, a dedicated vehicle outfitted with a bank of sensors first makes repeated passes scanning the roadway to be mapped. The data is then downloaded, with every square foot of the landscape pored over by both humans and computers to make sure that all-important real-world objects have been captured. This complete map gets loaded into the car's memory before a journey, and because it knows from the map about the location of many stationary objects, its computer—essentially a generic PC running Ubuntu Linux—can devote more of its energies to tracking moving objects, like other cars.
But the maps have problems, starting with the fact that the car can’t travel a single inch without one. Since maps are one of the engineering foundations of the Google car, before the company's vision for ubiquitous self-driving cars can be realized, all 4 million miles of U.S. public roads will be need to be mapped, plus driveways, off-road trails, and everywhere else you'd ever want to take the car. So far, only a few thousand miles of road have gotten the treatment, most of them around the company's headquarters in Mountain View, California. The company frequently says that its car has driven more than 700,000 miles safely, but those are the same few thousand mapped miles, driven over and over again.
...
Noting that the Google car might not be able to handle an unmapped traffic light might sound like a cynical game of "gotcha." But MIT roboticist John Leonard says it goes to the heart of why the Google car project is so daunting. "While the probability of a single driver encountering a newly installed traffic light is very low, the probability of at least one driver encountering one on a given day is very high," Leonard says. The list of these "rare" events is practically endless, said Leonard, who does not expect a full self-driving car in his lifetime (he’s 49).
The Google car will need a computer that can deal with anything the world throws at it. The mapping system isn’t the only problem. The Google car doesn’t know much about parking: It can’t currently find a space in a supermarket lot or multilevel garage. It can't consistently handle coned-off road construction sites, and its video cameras can sometimes be blinded by the sun when trying to detect the color of a traffic signal. Because it can't tell the difference between a big rock and a crumbled-up piece of newspaper, it will try to drive around both if it encounters either sitting in the middle of the road. (Google specifically confirmed these present shortcomings to me for the MIT Technology Review article.) Can the car currently "see" another vehicle's turn signals or brake lights? Can it tell the difference between the flashing lights on top of a tow truck and those on top of an ambulance? If it's driving past a school playground, and a ball rolls out into the street, will it know to be on special alert? (Google declined to respond to these additional questions when I posed them.) ...
Computer scientists have various names for the ability to synthesize and respond to this barrage of unpredictable information: "generalized intelligence,” "situational awareness,” "everyday common sense." It's been the dream of artificial intelligence researchers since the advent of computers. And it remains just that. "None of this reasoning will be inside computers anytime soon," says Raj Rajkumar, director of autonomous driving research at Carnegie-Mellon University, former home of both the current and prior directors of Google's car project. Rajkumar adds that the Detroit carmakers with whom he collaborates on autonomous vehicles believe that the prospect of a fully self-driving car arriving anytime soon is "pure science fiction."
Back when I was in banking, there was a term that got batted around quite a bit called speed-boating. The expression was derived from the way a fast-traveling boat can, for a while, outrun its own wake . As long as a certain speed is maintained, the boat will travel smoothly. However, if the boat suddenly slows down it can be swamped when its wake catches up with it.
Here's how the analogy worked in banking. When you are in the business of lending money, both regulators and investors like to keep track of how well you are doing at getting people to pay their loans back. To do this, they would look at the charge-off rate. At the risk of oversimplifying, this rate was basically the number of loans that went bad divided by the total number of accounts that were open during the period in question.
Obviously, if you booked an account and it went bad, this would add one to both the numerator and the denominator which would push your rate closer to 100%. So you would think that it would always be in the banker's best interest to avoid loans that are going to go bad.
The flaw, or at least the loophole, in this assumption is the fact that the ones are not added at the same time. Even in the extreme cases where the customers never make a payment, the loan is not considered bad for a certain interval, generally 90 days or more. If the customers make a few small payments, this could stretch out for six months or year.
Let's say I book an account that goes delinquent after one year. That was a bad deal for the bank – – it lost money due to that decision – – but for one year, having that account actually lowered the bank's charge-off rate. Eventually, of course, this will catch up with the bank, but the reckoning can be delayed if the bank continues to book these bad accounts at an increasing rate.
As with many of our posts, the moral of the story is that numbers don't always mean what you think they mean. You will often see someone pull out a statistic to settle an argument -- "How can you say the business model is unstable? See how long their charge-off rate is?" -- but without understanding the number and knowing its context, you can't really say anything meaningful with it.
Starting a new thread on Common Core over at the teaching blog. I'll be cross-posting the highlights but if you want to follow the whole thing, click on the link above.
Adrian: Once again we've got our friend from military intelligence. Can you tell us what you've found out about the enemy since you've been here?
Adrian as Gomer: We found out that we can't find them. They're out there, and we're having a major difficulty in finding the enemy.
Adrian: Well, what do you use to look for them?
Adrian as Gomer: Well, we ask people, 'Are you the enemy? And whoever says yes, we shoot them.
There is no profession anywhere in the country that has such astonishing rules. Good lord-- even if your manager at McDonalds decides you're not up to snuff, he doesn't blackball you from ever working in any fast food joint ever again! Yes, every profession has means of defrocking people who commit egregious and unpardonable offenses. But-- and I'm going to repeat this because I'm afraid your This Can't Be Real filter is keeping you from seeing the words that I'm typing-- Massachusetts proposes to take your license to teach away if you have a couple of low evaluations.
and
One version of the plan even allows for factoring in student evaluations of teachers; yes, teachers, your entire career can be hanging by a thread that dangles in front of an eight-year-old with scissors.
Wow. Isn't it a good thing that we are sure that this power will not be abused? I wish I could get Mark to give a perspective on this one, as I think it would have major implications.
Obviously this sort of scheme would only be workable for major offenses or where the barriers to entry to the profession are exceptionally low. Adam Smith pointed out that the less security that you give workers, the more expensive they get (for the same quality of worker). Now removing teacher tenure might or might not be a huge blow depending on the employment regulations in a state. There are professions (e.g. hedge fund trader) where people can move around a lot during a career. Teaching isn't well set up for this, but one can at least imagine making it work.
But losing one's license for bad test scores is a massive penalty. Heck, is it even the case that you'd lose a teaching license (as opposed to merely being fired) for tampering with the tests? Think carefully because employees will be able to work this one out . . .
A few months ago, Meredith Broussard ran a great expose of the role of textbook companies in the education reform movement (it's about other things as well, but that will get us started). I plan on discussing it more at length but the queue is getting long and I'd like to get this into the conversation as soon as possible.
The end result is that Philadelphia’s numbers simply don’t add up. Consider the eighth grade at Tilden Middle School in Southwest Philadelphia. According to district records, Tilden uses a reading curriculum called Elements of Literature, published by Houghton Mifflin. In 2012–2013, Tilden had 117 students in its eighth grade, but it only had 42 of these eighth-grade reading textbooks, according to the (admittedly flawed) district inventory system. Tilden’s eighth grade students largely failed the state standardized test: Their average reading score was 29.4 percent, compared with 57.9 percent districtwide.
One problem is that no one is keeping track of what these students need and what they actually have. Another problem is that there’s simply too little money in the education budget. The Elements of Literature textbook costs $114.75. However, in 2012–2013, Tilden (like every other middle school in Philadelphia) was only allocated $30.30 per student to buy books—and that amount, which was barely a quarter the price of one textbook, was supposed to cover every subject, not just one. My own calculations show that the average Philadelphia school had only 27 percent of the books required to teach its curriculum in 2012-2013, and it would have cost $68 million to pay for all the books schools need. Because the school district doesn’t collect comprehensive data on its textbook use, this calculation could be an overestimate—but more likely, it’s a significant underestimate.
This mentality also ruins human interaction. There is a tremendous variety of human beings within each political party. To judge human beings on political labels is to deny and ignore what is most important about them. It is to profoundly devalue them. That is the core sin of prejudice, whether it is racism or partyism.
There are some important points (and some questionable ones) made in this column. I'm not sure the 'ism' adds much to the discussion, but it's a topic that deserves more attention (and more pieces like this 2012 episode of This American Life). It's also a topic for another post.
For now let's stick with the con(firmation) artists thread. That's my somewhat cumbersome term for a school of journalists who get away with cliched, factually-challenged reporting because they appeal to the prejudices of their target audiences and, more importantly, of their peers and superiors. It is a group strongly associated with the New York Times and best exemplified by Brooks.
Much of Brooks' success comes from his ability to craft readable, scholarly sounding pieces that use statistics and anecdotes to reinforce class stereotypes, particularly those held by people on the top have toward people on the bottom. Frequently, when he could not find suitable support for his arguments, Brooks has used facts that aren't actually true. This might simply be the product of sloppiness, but it should be noted that the sloppiness always seems to occur in one direction.
At the risk of oversharpening, if you take the paragraph above and substitute the concept of class in for party, every criticism Brooks makes applies to much if not most of his own work. He has made a remarkably successful career out of understating the "tremendous variety of human beings" and judging them based on class and other crude demographic labels. What's worse, he continues to resort to statements that simply aren't true (like this claim about vaccines) in order to reinforce various stereotypes.
Does this behavior qualify as "the core sin of prejudice"? I'm not sure, but Brooks' partyism column certainly demonstrates a stunning lack of self-awareness.
Put simply, any teacher who wants his or her students to pass the tests has to give out books from the Big Three publishers. If you look at a textbook from one of these companies and look at the standardized tests written by the same company, even a third grader can see that many of the questions on the test are similar to the questions in the book. In fact, Pearson came under fire last year for using a passage on a standardized test that was taken verbatim from a Pearson textbook.
The issue often has as much to do with wording as it does with facts or figures. Consider this question from the 2009 PSSA, which asked third-grade students to write down an even number with three digits and then explain how they arrived at their answers. Here’s an example of a correct answer, taken from a testing supplement put out by the Pennsylvania Department of Education:
Here’s an example of a partially correct answer that earned the student just one point instead of two:
This second answer is correct, but the third-grade student lacked the specific conceptual underpinnings to explain why it was correct. TheEveryday Math curriculum happens to cover this rationale in detail, and the third-grade study guide instructs teachers to drill students on it: “What is one of the rules for odd and even factors and their products? How do you know that this rule is true?” A third-grader without a textbook can learn the difference between even and odd numbers, but she will find it hard to guess how the test-maker wants to see that difference explained.
One of these days, I need to post a nice long thread on targeted marketing, benefits vs. limitations, promise vs. hype. In the meantime, I'll collect as many examples as I can of targeting done well, targeting done badly and, in this case, of targeting not done at all.
In a comment to Ken Levine's previously mentioned traffic report prank.
In Canada, Closed Captioning is sponsored and announced during the broadcast. Frequently, these sponsors are new album releases - Katy Perry, Rihanna, Coldplay.
Assuming that the language of the captioning is the same as the language of the broadcast and is not used for translation (not an absolute given, I suppose, in Canada), think about the target demographic of captioning.
We’re all familiar with the critiques of standardized tests and other common measures used for high-stakes decisions. Recently, somebody in my circle has started going on about measures of “grit” and their predictive power. I am willing to believe that “grit” is an excellent predictor of all sorts of things. But I wonder if much of the predictive power of “grit” comes from the fact that these measures are currently low-stakes, so people have few incentives to game them.
I really think that this is the heart of the measurement problem. Insofar as there is a way to do better on a test, in a way that is less work than just be really good at it, then it is probable that much of your signal will be gaming. Studying the form of the question, for example, is likely to improve performance (by less confusion, if nothing else) but access to these approaches may vary by context.
Even worse, some of the test prep may have nothing to do with the underlying measure. So the score starts to measure things like "willingness to sacrifice learning time for test prep time".
This is a very good insight and likely to be eternally problematic in education.
I realize we've hammered this point quite a bit but, for their own good as well as the public's, charter schools have got to come up with an aggressive plan to deal with the self-dealing, price gouging and general looting that is becoming widespread in places like Michigan, Florida and Ohio.
The charter school Imagine School for the Arts is paying rent of nearly $1 million a year on a downtown building with the education funding it gets from the state, prompting criticism from a progressive advocacy group that studied charter-school finances around the state.
The complicated financial arrangement also involves a school-affiliated trust company spending more than $7 million last year to buy a building valued at less than $2 million.
The liberal advocacy group ProgressOhio attacked the size of the rent payments at charter schools operated in Toledo and other Ohio cities by Imagine Schools Monday as excessive. Imagine is a national for-profit educational management company.
According to ProgressOhio, Imagine’s subsidiary, Schoolhouse Finance, collected at least $14.4 million in public money last year for the company’s 17 Ohio schools. Of that, $8.9 million covered rent for long-term leases to Schoolhouse Finance. The $5.5 million balance went to pay “indirect costs” to Imagine to provide management services.
Both the tech and financial sector have embraced the idea that economic rewards are directly correlated to work and worth. It's a strange mixture of efficient market theorem and social Darwinism, often with more than a bit of Randianism.
Because rich people spend their money on useless stuff. Not far from where I live, there is a new house going up. It will be over 10,000 square feet when it is complete. 2,500 of those square feet will be a closet that has two separate floors, one for regular clothes and one for formal wear. If that is what you are spending your money on, then yes, I believe raising your taxes to fund education, infrastructure, and health spending is a net gain for society.
Don’t poor people spend money on stupid stuff? Of course they do. Isn’t the government an inefficient provider of some of these goods, like education? Maybe. But even if both those things are true, public investment and/or transfers to poor people will result in some net investment that I’m not currently getting from the mega-closet family. I’m happy to talk about alternative institutional settings that would ensure a greater proportion of the funds get spent on actual investments.
And,
Because I’m not afraid that some embattled, industrious core of “makers” will decide to “go Galt” and drop out of society, leaving the rest of us poor schleps to fend for ourselves. Oh, however will we figure out how to feed ourselves without hedge fund managers around to guide us?
I think that this points out that the notion that people deserve the actual income/wealth that they currently have, in some sort of fundamental way, is a true measure of worth/contribution. It is true that rich people do pay taxes, but consider all of the benefits they get? We have a whole society of laws devoted to reducing kidnapping, murder of near kin, and robbery, and the wealthy definitely benefit from this.
So I am not saying that the current level of taxes is too low, too high, or just right (on any specific segment of the market). I am saying that an open discussion should consider issues like "everybody spends money to satisfy desires and that these desires rarely pass the scrutiny of outside parties". Or that the whole idea of "going Galt" is often rather silly. In academics, we have many good people for every position. The marginal loss of any one person is sad but does not destroy the whole enterprise.
Similarly, I suspect that there are a number of possible hedge fund managers in the world who could do a relatively comparable job (at least insofar as society as a whole matters -- losing your superstar investor could be a private tragedy). Otherwise we'd expect the market to collapse when one of these key people dies of old age (and the market has been pretty robust to replacements via death so far).
