The Silent Cost of Conversation History in LLM Reasoning
This paper demonstrates that accumulating conversation history in multi-turn LLM interactions significantly degrades reasoning accuracy over time, a decline that standard consistency and coherence metrics fail to predict early, thereby necessitating external task state management and cross-session monitoring.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written by the author. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern digital world, artificial intelligence often acts as a tireless assistant, holding a conversation that stretches over dozens of exchanges. Imagine a logistics manager asking a computer to track inventory across a warehouse, or a financial planner updating a budget as new bills arrive. The computer remembers every previous message, building a long history of the conversation to understand the current question. This growing record is supposed to help the machine reason better, acting like a shared notebook between the human and the computer. However, there is a hidden danger in this approach. As the conversation gets longer, the computer must process not just the new question, but its own earlier answers, which might contain mistakes. If the machine makes an error early on, it can lock itself into that wrong idea, repeating the mistake consistently for the rest of the conversation without ever showing signs of confusion. The result is a system that sounds confident and fluent but is fundamentally wrong, and because the errors are consistent, the failure can remain invisible to the user.
Researchers set out to measure exactly how much this growing conversation history hurts the computer's ability to solve problems correctly. They tested this by running two different kinds of experiments. In the first, they took a set of math problems and split the information into small pieces, revealing them one by one. They asked the computer to solve the problem in two ways: once with a full record of its own previous answers, and once with a clean slate that only showed the new information. The results were clear and consistent. When the computer was allowed to keep its own earlier replies in its memory, its success rate dropped significantly. For one of the models tested, the chance of getting the final answer right fell by nearly thirty percentage points. For another, it dropped by about twenty-one points. The computer was not just getting confused; it was actively hindered by the very history it was supposed to use to help.
The second experiment pushed this idea to the extreme, simulating a conversation that lasted one hundred turns. This was a long, complex task where the computer had to track changes in a budget or inventory over time. The researchers carefully ensured that the difficulty of the questions stayed the same throughout the entire conversation, so they could be sure that any drop in performance was due to the length of the chat and not harder questions. They found that the computer's accuracy collapsed rapidly. Within just twenty turns, the ability to give the correct answer fell from about sixty percent down to ten percent. The machine was essentially failing almost every question, even though the questions themselves had not become harder. This collapse happened because the computer was building its new answers on top of a foundation of earlier errors, creating a chain of consistent but incorrect reasoning.
Crucially, the researchers also looked for a way to catch this failure before it happened. In a real-world system, you cannot always know the correct answer in advance, so you need a warning signal that tells you the computer is drifting off course. They tested several standard ways to measure the quality of the computer's thinking, such as checking if its answers were consistent with each other or if its logic flowed smoothly. They hoped that as the computer started to fail, these quality scores would drop, sounding an alarm. Instead, they found the opposite. As the accuracy plummeted, the measures of consistency and logical flow actually stayed high or even improved. The computer became more stubbornly consistent in its wrongness. Its answers were perfectly coherent with each other, just perfectly wrong. The usual tools for spotting trouble were completely silent, offering no warning that the system was failing.
The study also tested whether special computer programs designed to detect changes in behavior could spot the problem. These programs are like smoke detectors for data, looking for sudden shifts that indicate something is wrong. However, the researchers found that these detectors were too slow to be useful. By the time the detectors finally sounded an alarm, the computer had already been failing for a long time. The decline in performance started so early in the conversation that the detectors could not establish a baseline of normal behavior to compare against. Even when the researchers tried to fix this by looking at many conversations at once, the warning signals remained weak and unreliable. The only method that worked was to remove the conversation history entirely. Instead of feeding the computer its own long transcript, the researchers gave it a short, verified summary of the current state, written by a separate, reliable system. When they did this, the computer's accuracy stayed high and stable throughout the entire hundred-turn conversation, proving that the problem was the history itself, not the task.
The findings suggest that for any system relying on artificial intelligence to manage complex, long-term tasks, keeping a full transcript of the conversation is a liability. The machine does not get smarter by remembering its own mistakes; it gets trapped by them. The researchers conclude that the best approach is to keep the task state, such as a budget total or an inventory count, in a separate, verified record outside of the conversation. This record should be sent to the computer fresh for every new question, rather than letting the computer reconstruct the state from its own flawed memory. While the computer's ability to hold a long conversation is impressive, this study shows that for serious reasoning tasks, less history is often more accurate. The silence of the warning systems serves as a stark reminder that a fluent, consistent answer is not always a correct one, and that in the world of artificial intelligence, the most reliable memory is sometimes the one that forgets the past.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.