💬 NLP

Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control

This paper introduces RAD (Routing Agreement Decoding), a novel inference strategy that leverages the distinct Mixture-of-Experts routing patterns of identical tokens to select high-quality reasoning trajectories without relying on answer string parsing or voting, thereby enabling effective pass@1 selection for code and agentic tasks where traditional majority voting fails.

Original authors: Kang Chen, Minshen Yu, Junjie Nian, Yaoning Wang, Yixin Cao, Yugang Jiang

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kang Chen, Minshen Yu, Junjie Nian, Yaoning Wang, Yixin Cao, Yugang Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Is a "Hello" Always the Same "Hello"?

Imagine you are listening to a choir of 100 different singers (the "Experts" inside a large AI model). When the choir sings the word "Hello," you might assume they are all singing it the exact same way, using the same vocal techniques and emotions.

The paper asks: If the AI outputs the exact same word (token) in two different situations, does it mean the internal "machinery" that produced that word was also the same?

The Answer: No.

Even if the AI writes the exact same word (like "Hello" or "The answer is 42"), the specific group of "singers" (experts) inside the model that produced it can be completely different depending on the context. One "Hello" might be sung by a group of experts thinking about math, while another "Hello" is sung by a group thinking about coding. The word is the same, but the internal state is different.

The Problem: How Do We Pick the Right Answer?

Usually, when we want an AI to solve a hard problem (like a math puzzle or a coding bug), we ask it to try 64 different times (64 "rollouts"). Then, we look at the final answers. If 50 of them say "42" and 14 say "43," we pick "42" because it's the majority vote.

The Catch: This "voting on the final answer" works great for math (where the answer is a clear number). But it breaks down for:

Code: Two programs can do the exact same thing but look totally different (different variable names, different formatting). You can't just count how many times the word "print" appears.
Agents: Sometimes the AI is trying to fix a software bug. The "answer" is a patch of code. There is no single "correct string" to vote on; the code is unique to every attempt.

We need a way to pick the best attempt without reading or comparing the final answers.

The Solution: RAD (Routing Agreement Decoding)

The authors discovered a secret signal hidden inside the AI: The Router.

Think of the AI as a massive office building with thousands of specialized workers (experts). Before a worker does any task, a Router (a manager) decides which team of workers gets the job.

The Discovery: The paper found that when the AI is about to write the start of an answer (like the \boxed{ symbol in math or the triple backticks ``` in code), the Router's decision pattern reveals a lot about the quality of the answer.
The Analogy: Imagine you are trying to guess which team in a sports tournament will win. Instead of waiting for the final score (which you can't see yet), you look at which players the coach chose to start the game.
- If the coach picks the same "dream team" lineup for 50 different games, those games are likely going to end up with the same result.
- If the coaches pick random, mismatched lineups, the results will be all over the place.

RAD works like this:

It generates 64 different attempts at a problem.
It ignores the final answers completely. It doesn't read them.
It looks at the Router's lineup (which experts were chosen) right at the moment the answer begins.
It asks: "Which 64 attempts had the most similar lineups?"
It picks the attempt that belongs to the largest group of similar lineups.

If 50 attempts all used the same "expert team" to start their answer, RAD assumes that group is the most confident and likely to be correct, even without knowing what the answer actually is.

Why This is a Big Deal

It Works Where Voting Fails: On coding tasks, where you can't just count matching words, RAD still works. It picks the best code patch by looking at the "internal team" that wrote it, not the code itself.
It's Fast and Simple: It doesn't need to understand the meaning of the code or math. It just looks at the pattern of the internal "switches" being flipped.
It's Not a Magic Truth Detector: The paper is honest about this. If 50 people all agree on the wrong answer (a "dense wrong basin"), RAD will pick that wrong answer too. It picks the most popular path, not necessarily the true path. It's a "consensus signal," not a "truth verifier."

Summary in One Sentence

The paper shows that even if an AI says the same word twice, the internal "team" that said it might be different; by picking the answer that comes from the most consistent "team lineup" at the start of the response, we can choose the best result without ever needing to read the final answer.

or<|channel|>final<|message|>). * *Delimiter anchors:* Fixed answer-opening delimiters (e.g., \boxed{` for math, code fences for code).
2. Readout: Extracts the sparse MoE routing vectors (selected expert IDs and post-softmax weights) within a fixed window $W$ following the anchor. The headline setting uses delimiter@16 (a 16-token window starting at the delimiter).
3. Representation: Averages the routing vectors over the window to create a per-rollout route vector $z_i$ .
4. Selection: Computes pairwise Weighted-Jaccard (WJ) similarities between all route vectors. It calculates a local agreement density $q_i$ for each rollout based on its $K$ -nearest neighbors (typically $K=10$ ) and selects the rollout with the highest density (the center of the densest routing basin).

Key Design Choices

Answer-String-Free: The selector never parses, normalizes, compares, or votes over the final answer string. The answer is only read after the routing-based selection is made for evaluation.
Agentic Adaptation: For agentic tasks (e.g., SWE-bench) where no answer string exists, RAD uses a boundary anchor (thinking-to-action transition) and a binary Jaccard fingerprint of activated (layer, expert) pairs, maintaining the same density-selection principle.
Fusion (RAD+DC): The method can be combined with DeepConf's scalar confidence scores (filtering for the top 50% confidence rollouts before applying RAD density) to handle cases where correct paths are rare.

3. Key Findings and Analysis

Descriptive Insight: Token Identity $\neq$ Router State

The paper demonstrates that holding the emitted token ID fixed does not fix the internal router state.

Experiment I: At a fixed token ID (e.g., the Qwen EOS token or a specific boundary marker), the experts generating that token still separate problem identity and reasoning-effort modes (Low/Medium/High).
Implication: The routing state contains "residual" information about the upstream context and task family that the surface token ID alone does not encode.

Operational Insight: Routing Neighborhoods Align with Answer Basins

Experiment II: Routing neighborhoods (clusters of similar routing vectors) align with "answer basins" (groups of rollouts sharing the same normalized answer).
Strongest Signal: This alignment is strongest at delimiter windows (specifically delimiter@16), where the routing state reflects the model's commitment to a specific answer basin.
Correlation: There is a strong correlation between the purity of the routing neighborhood (how many neighbors share the same answer) and the accuracy of the selector. A dense routing basin often approximates a dense answer basin.

4. Experimental Results

The authors evaluated RAD across 10 sparse-MoE configurations (gpt-oss 20B/120B, Qwen3-30B, Qwen3-Next-80B) and 6 datasets (Math: AIME, BRUMO, HMMT; Science: GPQA; Code: LiveCodeBench; Agents: SWE-bench Verified).

Performance on Well-Posed Tasks (Math & GPQA)

RAD is on par with textual Majority voting (self-consistency) where string voting is well-defined.
Results: Pooled problem-weighted average: Majority 73.6% vs. RAD 73.9% / RAD+DC 74.2%.
The difference is small and statistically non-significant in natural pools, but RAD achieves this without reading any answer strings.

Performance on Ill-Defined Tasks (Code & Agents)

Code Generation (LiveCodeBench): Exact-string voting is degenerate (outputs are near-unique). RAD provides a direct pass@1 selection, outperforming the random baseline (Avg@64) by significant margins (e.g., +9.7 pp for gpt-oss-20B).
Agentic Selection (SWE-bench Verified): In a best-of-16 patch selection task where patches have no answer string to vote on, RAD improves selection accuracy over random by +4.7 to +5.0 percentage points on the decidable subset.

Robustness

Kernel: Binary Jaccard and Weighted Jaccard are statistically indistinguishable in performance.
Depth: The signal is redundant across layers; shallow layers alone capture most of the signal.
Access: Even strict "marker-only" readouts (reading only the delimiter token, not the following 16 tokens) recover most of the performance lift, confirming the signal exists before answer content is generated.

5. Claims and Significance

The paper positions RAD not as a verifier (it does not predict truth) but as an agreement signal.

Epistemic Status: Like Majority voting, RAD estimates "shared commitment." A dense routing basin can still be wrong if the model converges on a shared error. However, it provides a mechanism to select the most populated basin without relying on surface text.
Interface Value: The primary contribution is a unified, answer-string-free interface for multi-rollout selection. It works where string voting fails (code, agents) and performs competitively where string voting works (math, science).
Scalability: The method scales with rollout count (improving as $N$ increases) and is computationally efficient, adding only ~7% overhead during generation for routing capture and running selection in real-time on a single CPU thread.

Conclusion: The paper establishes that MoE routing states contain a "white-box" agreement signal that is robust to token identity variations and aligns with answer basins. RAD operationalizes this to enable reliable multi-rollout selection across diverse modalities without the fragility of text-based aggregation.