I was reading this blog post by Andrew Gelman on a test that is 100% accurate for alzheimer's disease. Following the initial post up, it appears that the test has only 64% specificity.
But the feature of this discussion that I find the most interesting is that the decision to choose between sensitivity and specificity is a judgment that people seem to be very poor at. Consider pain medication. If you want to make sure that everyone in serious pain gets appropriate pain control than some fraudsters will get illicit narcotics. Alternatively, if you make the screen tight that nobody is able to obtain narcotics via "fake pain" then some real cases will be undertreated.
We see the same thing with releasing people from prison. Even if former prisoners only committed crimes at the rate of the general population, at least some crimes could be prevented by a tougher release policy. Of course, this line of reasoning leads to absurd conclusions -- we could completely eliminate adult crime by jailing everyone for life on their eighteenth birthday. We see the same thing in the Ray Fisman argument for getting rid of 80% of teachers during probation – it is so important not to make a mistake and keep an inferior teacher that we should fail to hire many good teachers just to make sure we have no sub-standard ones.
But people don't seem to like to make these trade-offs. In the case of the test for Alzheimer's disease, the authors could have been a lot more specific if they were willing to give up sensitivity. But, for some reason, people seem to prefer to end up at one extreme of a scale rather than the middle (where the value of the test is maximized).
It's a phenomenon that I wish I understood better.
The Accreditation Conundrum
2 hours ago