Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations
This paper introduces a graded, multi-family stress-testing framework that reveals how language models' reasoning stability and failure points vary across specific perturbation types and severity levels, exposing critical weaknesses in handling conflicting instructions and impossible premises that standard accuracy metrics overlook.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of artificial intelligence, a computer program that reads and writes language is often judged by a single number: its accuracy score. This score tells us how often the machine gets the right answer on a standard test. But this single number hides a great deal about how the machine actually thinks. It does not show us what happens when the question is asked in a different way, or when the instructions are slightly confusing, or when the question itself contains a trick that makes it impossible to answer. Just as a bridge might hold up under heavy traffic but collapse if a single bolt is loose, a language model might solve a math problem perfectly in one format but fail completely if the words are rearranged or if a distracting sentence is added. Understanding these hidden weaknesses is crucial because real-world problems are rarely clean or perfectly phrased. If we rely on these machines for important decisions, we need to know not just if they can solve a problem, but how they behave when the conditions are difficult or when the problem itself is broken.
A researcher set out to map these hidden behaviors by stress-testing four different language models. Instead of asking the models a single question and recording a pass or fail, they created a vast laboratory of 4,473 variations of 100 simple grade-school math problems. They took each problem and subjected it to seven different types of pressure, ranging from mild changes to severe distortions. Some changes were like a gentle breeze, such as rewriting the sentence to sound more formal or adding a few typos. Others were like a gale force wind, such as inserting contradictory instructions that told the model to ignore the math and give a different answer, or adding a long list of irrelevant facts that had nothing to do with the calculation. The researcher also included a special category of questions that were impossible to answer, such as those based on facts that do not exist or missing essential numbers, to see if the models would admit they did not know the answer or if they would confidently make up a number.
The study revealed that the models did not fail in a uniform way. While the strongest models could handle heavy changes to the wording or the layout of the text without losing their grip, they all stumbled when faced with specific types of confusion. Every single model, regardless of how advanced it was, eventually broke down when the instructions conflicted with each other. If the prompt told the model to solve a problem but then added a command to ignore the problem, the models tended to get confused and fail. Even more telling was how the models handled questions that could not be answered. When the researcher presented questions based on impossible premises, such as asking for the result of a calculation that relied on a fictional event, the models almost always tried to solve them anyway. They calculated a number and presented it as a fact, rather than stopping to say that the question made no sense. This happened even for the most capable model in the test, which otherwise performed very well.
The researcher found that the point at which a model failed depended entirely on the type of trouble it faced. One model might be very strong against typos but weak against long, distracting paragraphs, while another might handle distractions well but collapse under conflicting commands. This means that a single overall score is not enough to describe a model's reliability. The study showed that the ability to recognize when a question is unanswerable is uneven. The models were good at refusing to answer when information was clearly missing or when evidence was obviously fake, but they were very poor at spotting questions built on impossible premises. Instead of flagging the error, they would proceed with the math and return a confident, but wrong, result. These failures were invisible in standard testing because the models were never asked to deal with these specific kinds of stress in a graded, systematic way.
By measuring how performance dropped as the difficulty increased, the researcher created a detailed profile of each model's strengths and weaknesses. They found that the strongest models were robust against surface-level changes, such as rephrasing or formatting, but shared a common fragility when it came to conflicting instructions and impossible scenarios. This suggests that while these machines have become very good at following patterns, they still struggle to maintain their task when the rules are twisted or when the problem itself is flawed. The study concludes that to truly understand how these models will behave in the real world, we must look beyond a single accuracy number and examine how they degrade under pressure. The results show that the path to more reliable artificial intelligence requires not just better performance on standard tests, but a deeper ability to recognize when a question cannot be answered and to stay focused even when the instructions are contradictory.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.