Why a predictably breakable rope is better than an unbreakable rope

Short answer: there is no such thing as an unbreakable rope.


There's an old story about an isolated monastery located high on the side of an unclimbable cliff. The only access to the monastery was by way of a basket that was hold up the side of the cliff on a single rope. One day a pilgrim who was climbing into the basket noticed that the rope looked old and parade. He asked the monk "when do you replace the rope?"

The monk replied "when it breaks."

If we generalize a bit, this becomes a useful analogy. We have a case where there is great cost associated with avoidable failure, but where there are also nontrivial costs associated with caution.

One common but probably misguided response to the situation is to buy a better rope i.e. come up with a system that is less likely to fail. If you have a shoddy system with lots of room for cheap and easy improvement, this approach makes a great deal of sense. If, on the other hand, you have already made all of the obvious and inexpensive upgrades, it probably makes more sense from a cost benefit perspective to start focusing on the question of when you replace the rope.

You frequently see this question coming up in connection to proxy variables. Particularly in the social sciences, researchers are constantly required to substitute an easily measured variable for the actual factor of interest. If we start with a "good rope" (a well-chosen proxy) then it will, under most circumstances, correlate strongly with the thing we are actually interested in.

There are plenty of "bad ropes" out there, proxies that have only weak relationships with the variables of interest even under the best circumstances, but that is a topic for another post. The disagreement here is with the otherwise responsible statisticians who make an effort to find the best possible proxy but who then do not spend enough time thinking about what happens when the rope breaks.

A few years ago, while I was doing risk models for a large bank, I found myself caught in a heated debate. We had a very good direct measure of how close people were to maxing out their line of credit. Unfortunately, this was also an expensive variable, so it was proposed that we substitute another, less direct measure. The argument for the substitution was that there was an extremely high correlation between the two variables. The counter argument put forward by most of the more experienced statisticians was that while this was true, that correlation tended to break down in extreme cases, particularly those where a person was about to go bad on all of their debts . Since the purpose of the model was to predict when people were about to default on their loans, this was a really unfortunate time for the relationship to fall apart.