Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form
This paper challenges the "admission gate" hypothesis for latent variable access in language models, demonstrating instead that demand-driven attention mechanisms gather information within a specific mid-depth window to render variables verbally reportable, a process that is distinct from the actual utility of the variable for the final answer.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Inside the vast, silent architecture of a modern language model, a hidden world of information exists long before the machine speaks a single word. These models are trained on oceans of text, learning to predict what comes next by weaving together patterns of language, logic, and fact. When a model is asked a question, it does not simply search a database; it constructs an internal representation of the answer, a latent variable that holds the specific meaning needed for the task. For years, researchers have wondered how this hidden information becomes visible. The prevailing intuition suggested a simple mechanism: the information is always there, sitting in the model's internal memory, waiting for a gate to open and let it out. In this view, the model acts like a librarian who already has the book on the shelf and simply decides to hand it over when asked.
A researcher at the University of California, Santa Cruz, set out to test this idea of a "gate" using a large language model called Qwen3.6. They designed a series of experiments where the model was given the exact same story but asked different questions about it. In some cases, the model needed to use a specific piece of information from the story to solve a puzzle; in other cases, it needed to report that information directly; and in a control case, it was asked a question that required no knowledge of the story at all. By carefully measuring the model's internal state at every step of its processing, they were able to see exactly when and how the hidden information became available to the final answer.
What they found overturned the simple gate theory. The hidden information was not sitting in the model's memory waiting to be unlocked. Instead, the model actively gathered the information from a distant part of its internal stream and transported it to the specific spot where the answer would be formed. This gathering process did not happen everywhere or all the time. It occurred within a very specific window in the middle of the model's processing layers, roughly between the 36th and 42nd layers out of 64. Below this window, the information was present but could not survive the journey to the end; above it, the information was already gone or destroyed. The model only performed this gathering when the task actually demanded it. When the model was asked a question that required no knowledge of the story, the gathering process barely happened at all, concentrating seven times less effort than when the information was needed.
They discovered that this transport was carried out almost entirely by the model's attention mechanism, the part of the system that decides which parts of the input to focus on. The other major component, the feed-forward network which acts like a local processor, did not help move the information; in fact, in the middle layers where the gathering happened, it actively worked against the transport. The information was not being unmasked from a static location; it was being physically moved, like a courier carrying a package from a warehouse to a delivery truck. This movement was so precise that the model could concentrate the information in the middle layers with an intensity seventeen times greater than anywhere else in the system.
Crucially, they showed that the amount of information visible in the model's internal state did not always match how useful that information was for the final answer. They found that three different parts of the model could shift the internal signal by nearly the same amount, yet their actual impact on the final answer varied by a factor of seven. This means that simply seeing a strong signal in the model's internal state is not enough to know if the model is actually using that information correctly. The size of the signal does not tell the whole story of how the model thinks.
The study also revealed that this gathering window is a fundamental feature of the model's design, appearing at the same relative depth in different types of models, including a dense architecture with 62 layers. They confirmed that the information was not just present but causally necessary; if they blocked the transport in the middle layers, the model failed to answer correctly. However, they also noted that this gathering was necessary but not sufficient on its own; the model still needed to perform other calculations to produce the final answer. The work suggests that the model does not have a static workspace where information sits waiting to be admitted. Instead, it dynamically assembles the pieces it needs, moving them into place only when the task requires it, a process that is far more active and selective than previously imagined.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.