A Mechanistic Analysis of In-Context Learning and Out-of-Context Hallucination
This paper proposes a unified mechanistic framework explaining both in-context learning and out-of-context hallucination as outcomes of how role information is internally represented and aggregated, demonstrating that successful adherence to rules versus hallucination depends on the magnitude and coherence of role span contributions at key decision points.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer
Large language models are the engines behind a new generation of artificial intelligence, capable of reading a prompt and generating human-like text on demand. One of their most celebrated abilities is in-context learning, where a model can infer a specific rule from the examples provided in a prompt and follow it faithfully. If a user asks the model to summarize a list of records as a JSON object, the model can learn that format and apply it perfectly. Yet, these same systems suffer from a frustrating flaw known as hallucination. Sometimes, when asked to answer a question based on provided evidence, the model ignores the facts right in front of it and invents details that are not supported by the text. This creates a puzzling contradiction: how can the same machine be so obedient to a rule it just learned, yet so careless with the facts it was given? For years, researchers have studied these two behaviors separately, treating them as distinct phenomena. But this separation has left a gap in our understanding of why the model succeeds in one case and fails in the other.
A team of researchers at the National University of Singapore has now bridged this gap by looking at the internal mechanics of these models. They propose that both the successful following of rules and the failure to stick to facts rely on the same underlying process. The key to their discovery lies in how the model processes the different parts of a prompt. In any given prompt, some text acts as a rule, telling the model how to structure its answer, while other text acts as evidence, providing the specific facts to be used. The researchers found that the model does indeed "understand" the difference between these roles internally. It creates a specific, organized pattern of activity in its neural network to represent whether it is looking at a rule or a piece of evidence. However, simply having this internal representation is not enough to guarantee a correct answer. The model can recognize the role of the text without actually letting that recognition control its final output.
To uncover how this works, the researchers developed a way to measure the influence of specific parts of the prompt on the model's decisions. They focused on the "role span," which is the specific section of text containing either the rule or the evidence. By comparing the model's behavior when this text is present versus when it is replaced with different content, they could isolate exactly how much that specific text was pushing the model toward a certain answer. They discovered that for the model to follow a rule or use evidence correctly, the influence from that text must be strong enough and must point in the right direction consistently. It is not enough for the model to just "see" the rule; the mathematical signal from that rule must be large enough to overcome other competing signals in the model's brain, and it must align with the direction of the correct answer at every step of the generation process.
The study reveals that the reason models follow rules so well but often fail with facts comes down to where and how this influence is applied. When a model is following a rule, such as formatting an answer as a list, the influence from the rule text needs to be sustained across many different decisions. The model must repeatedly choose the right structure, and the rule's signal must remain strong enough at each of these steps to keep the model on track. In contrast, when a model is supposed to use evidence, the critical moment is much more specific. The influence from the evidence text must be powerful enough at the exact moment the model decides on a specific fact. If the signal from the evidence is too weak at that precise instant, or if it gets canceled out by other signals, the model will drift and invent a fact instead of using the one provided.
The researchers tested this theory by intervening directly in the model's calculations. They found that when they artificially weakened the signal coming from the rule text, the model frequently forgot the format it was supposed to follow. Similarly, when they reduced the influence of the evidence text, the model stopped using the provided facts and started hallucinating. Perhaps most revealingly, they showed that this failure is not always permanent. In cases where the model initially failed to use the evidence, the researchers were able to reverse the mistake by artificially boosting the signal from the evidence text. When the support for the correct fact was made strong enough to overcome the model's initial hesitation, it successfully switched to the right answer. This suggests that the model often has the correct information available internally, but it fails to act on it because the signal is not strong enough to win the internal competition at the moment of decision.
This work offers a unified explanation for two seemingly opposite behaviors. It suggests that the difference between a model that follows instructions perfectly and one that lies about the facts is not a difference in how it understands the prompt, but a difference in how the influence of that prompt is sustained during the generation process. For rules, the influence must be a steady, persistent force guiding the structure of the response. For facts, the influence must be a sharp, decisive push at the exact moment a specific value is chosen. The study concludes that the model's reliability depends on whether the internal signal from the prompt is strong enough and organized correctly to win these critical moments of choice. By understanding this mechanism, we gain a clearer picture of why these powerful tools sometimes work like a perfect assistant and other times like a confident liar, and how the same internal machinery is responsible for both outcomes.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.