Beyond the Trace: Coupling an Interpretable Reasoning-State Readout to Native MoE Routing
This paper introduces a two-level internal readout framework that distills latent reasoning states into a 64-axis semantic frame (J64) and reconstructs it via native expert-routing statistics (R64), enabling interpretable, low-overhead test-time decisions that significantly improve reasoning accuracy and allow for targeted mechanism edits to correct model behaviors.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a computer program designed to solve difficult problems speaks its thoughts aloud, we usually only see the final sentences it chooses to write. These spoken thoughts, often called reasoning traces, are like a transcript of a conversation where the speaker has edited out every hesitation, every false start, and every internal debate. For researchers trying to understand how these artificial minds work, this transcript is frustratingly incomplete. It shows the destination but hides the journey. The real question is whether the computer's internal state—the hidden machinery humming beneath the words—holds clues about whether it is on the right track long before it finishes its answer. If we could see that hidden state, we might be able to steer the computer away from mistakes while it is still thinking, rather than just discarding a failed attempt after the fact.
A team of researchers at Fudan University and the Shanghai Innovation Institute has built a new way to look inside these thinking machines. They focused on a specific type of advanced computer model that uses a "mixture of experts" architecture. Imagine a large office where, for every task, a manager automatically assigns the work to a small team of specialized workers. The computer does something similar: for every word it generates, it activates a specific group of internal specialists. The researchers realized that while the computer writes down its thoughts, it also keeps a detailed log of which specialists it used and how much it relied on them. This log is usually just a technical record of efficiency, but the team discovered it could be translated into a readable map of the computer's reasoning process.
The researchers first created a new kind of dashboard, which they call a semantic frame. They trained this dashboard to translate the computer's raw, hidden internal signals into sixty-four distinct categories of thought. These categories are not just random labels; they represent concrete concepts like "caution," "arithmetic," "checking constraints," or "considering options." Crucially, this dashboard can detect these concepts even when the computer never actually writes the corresponding words. For example, the computer might be silently struggling with a complex constraint, and while it writes nothing about it, the dashboard lights up to show that the "constraint" concept is active. This revealed a hidden layer of the thinking process that the written text completely missed. In their tests, this internal view provided a significantly clearer signal about whether a solution would succeed than simply counting the words the computer had already written.
The team then faced a practical challenge: reading these hidden signals usually requires stopping the computer and replaying its entire thought process, which is too slow for real-world use. They solved this by building a second, lightweight version of the dashboard that reads only the log of specialist assignments. This new tool, which they call a routing proxy, acts as a fast, low-cost substitute. It reconstructs the same sixty-four categories of thought directly from the computer's internal routing log. The researchers found that this proxy was remarkably accurate, capturing between 69% and 86% of the detailed information found in the full, slow version. On one of their test models, it preserved nearly all of the predictive power of the original, proving that the computer's own internal traffic log contains a faithful record of its reasoning state.
With these tools in hand, the researchers tested how they could improve the computer's performance at the moment it was solving a problem. They used the dashboard to make two types of decisions. First, after the computer generated many different attempts at a single problem, the dashboard helped them pick the single best attempt more often than random chance or simple text analysis could. In cases where standard voting methods failed to find a correct answer, this internal view helped rescue the correct solution in about 17% of those difficult cases. Second, they used the dashboard to stop bad attempts early. As the computer generated text, the dashboard monitored its internal state in real-time. If the dashboard detected that the computer was drifting into a dead end or getting stuck in a loop of guessing, the system stopped that attempt immediately and started a fresh one. This "stop and resample" strategy improved the final accuracy of the computer by up to 5.9 percentage points compared to letting it run without intervention.
Perhaps the most striking finding was that they could use this understanding to fix the computer's behavior directly. By identifying which specific internal specialists were responsible for a particular type of error, the researchers could make tiny adjustments to how the computer assigned work. In one instance, they identified a group of specialists that kept the computer stuck in a cycle of guessing numbers. By slightly suppressing the activity of that specific group, they forced the computer to switch to a more precise method of symbolic calculation, which allowed it to solve a problem it had previously failed. This demonstrated that the dashboard did not just describe the computer's state; it identified the exact mechanical levers that controlled its reasoning.
The work suggests that the path to better artificial intelligence may not lie in making models larger or training them on more data, but in learning to read the silent signals they generate while they think. The researchers showed that the internal routing of a model is not just a hidden technical detail, but a rich source of information about what the model is actually doing. By translating these signals into a readable format, they turned a black box into a transparent process. This approach allows us to intervene while the model is thinking, correcting its course before it reaches a wrong answer. While the study focused on mathematics problems, the method offers a new way to observe and guide the reasoning of complex systems, turning the invisible process of machine thought into something we can see, measure, and improve.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.