Selective Prediction Under Distribution Shift: The Failure of Threshold Transfer
This study demonstrates that standard threshold-based selective prediction fails under distribution shift because the optimal cutoff determined on clean data does not transfer to corrupted or real-world data, necessitating the use of new labeled data to establish statistically valid stopping rules.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a computer program trained to recognize objects in photographs. In a perfect world, this program would always be right. But the real world is messy. A camera might get a smudge of rain, a photo might be taken in dim light, or an object might be viewed from a strange angle. When the pictures the computer sees start to look different from the ones it studied during training, the program often gets confused. It might still give an answer, but that answer is likely to be wrong. To handle this, researchers have developed a safety valve: they allow the computer to say, "I do not know." If the program is not confident enough, it stays silent instead of guessing. This is called selective prediction. The goal is simple: set a rule that tells the computer when to speak and when to stay quiet, ensuring that whenever it does speak, it is right most of the time.
For years, the standard way to set this rule has been to test the computer on clean, perfect pictures. Researchers pick a confidence level that works for those clean pictures, such as a setting that guarantees the computer is wrong only 5% of the time when it speaks. They then take that same setting and ship it out to the real world, assuming it will hold up even when the pictures get blurry or distorted. A new study by Shayam Ahmad at C.V. Raman Global University challenges this assumption. The research asks a direct question: does a rule that works on clean data still work when the data changes? The answer is a resounding no. The study finds that the rule breaks down completely under real-world conditions, often leading to error rates far higher than promised, while the computer continues to speak confidently on pictures it should have rejected.
The researchers tested this idea using a standard set of images and two different types of computer vision models. They first trained the models on clean pictures and then subjected them to various types of damage, such as adding noise, blurring the image, or changing the colors. They also tested the models on a new set of real-world photos that looked slightly different from the training data, mimicking the kind of slow drift that happens when a system is used in the field for months. In every case, they applied the same rule that had been chosen on the clean data. The results were stark. When the researchers set a target of 5% error, the actual error rate on the damaged images jumped to between 42% and 45% in the worst cases. In other words, the computer was wrong nearly half the time, even though the rule promised it would be wrong only 5% of the time. The problem got even worse when they tried to be stricter. If they aimed for a 1% error rate, the actual error became nearly 20 times higher than the promise.
One might wonder if the computer simply got bad at ranking its own confidence. Perhaps it was sorting the pictures incorrectly, putting the easy ones at the top and the hard ones at the bottom, but failing to draw the line in the right place. To test this, the researchers separated the problem into two parts. The first part is the ranking: how well the computer orders the pictures from "most sure" to "least sure." The second part is the stopping rule: where to draw the line to decide when to stop answering. They found that the ranking was actually quite good. Four different ways of measuring confidence all produced nearly the same order. The computer knew which pictures were harder than others. The failure was entirely in the second part. The line drawn on the clean data was simply in the wrong place for the messy data. It let in far too many difficult pictures, accepting them as if they were easy, which caused the error rate to explode.
The study also looked at whether existing tricks could fix this without needing new information. Some methods try to guess the error rate of the new data without seeing the correct answers, hoping to adjust the rule automatically. The researchers tested two of these published methods. Neither worked. They reduced the error slightly but did not fix the problem, and both performed worse than a very simple rule that just guessed based on average confidence. The only way to truly fix the problem was to use a small amount of new, labeled data from the messy environment. By showing the computer a few hundred or a few thousand of the new pictures with their correct answers, the researchers could find a new stopping rule that actually met the safety target. However, this fix comes with a heavy price. Under mild changes in the data, they needed about 800 labeled examples to get a reliable rule. Under strong corruption, they needed more than 4,000. This creates a difficult situation: the more the data changes, the more labeled examples are needed to stay safe, yet labeled examples are hardest to get when the data is changing the most.
The implications of this finding are practical and immediate. It suggests that simply taking a rule designed for perfect conditions and applying it to the real world is dangerous. The promise of safety is an illusion if the data has shifted. The researchers conclude that if a system operator cannot afford to label new data from the field, they should not make a specific promise about error rates at all. Stating a target that cannot be kept is worse than stating none, because it gives a false sense of security. The study does not offer a magic bullet that solves the problem without cost, but it does clear away a major misconception. It shows that the failure is not in the computer's ability to judge difficulty, but in the rigid rule used to decide when to speak. The path forward requires acknowledging that the rules of the game change when the environment changes, and that keeping a promise of safety in a shifting world requires paying a price in new information.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.