💬 NLP

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

This paper reveals that standard top- $k$ routers in Mixture-of-Experts models often fail to select optimal expert paths for tokens requiring complex reasoning, but a minimal update to the router alone can significantly improve performance on challenging benchmarks by correcting these misallocations without retraining the experts.

Original authors: Youngsik Yoon, Siwei Wang, Wei Chen, Jungseul Ok

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Youngsik Yoon, Siwei Wang, Wei Chen, Jungseul Ok

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Expert Team" Problem

Imagine a massive, super-smart language model (like a giant AI brain) is built not as one single brain, but as a team of 100 specialized experts.

The Experts: Some are math whizzes, some are poets, some are historians, and some are logic puzzle solvers.
The Router: There is a "Manager" (called a router) whose only job is to look at every word the AI is about to say and decide which top 8 experts should work on that specific word.

The goal is efficiency: instead of waking up all 100 experts for every word (which would be slow and expensive), the Manager only wakes up the 8 most relevant ones. This is called a Mixture-of-Experts (MoE) model.

The Discovery: The Manager Gets Lazy on Hard Problems

The researchers asked a simple question: "Is the Manager actually picking the best 8 experts for every word?"

To find out, they didn't just trust the Manager. They froze the entire AI team in place (so no one could learn anything new) and ran a simulation:

They took a word the AI was about to say.
They asked the Manager: "Who did you pick?"
Then, they ignored the Manager and randomly picked other groups of 8 experts from the same pool to see if any other group could have done a better job on that specific word.

The Results:

On Easy Words (Confident Tokens): When the AI is talking about simple things (like "The sky is blue"), the Manager is great. The 8 experts it picks are almost always the best ones. The other groups of experts do about the same job.
On Hard Words (Fragile Tokens): When the AI is solving a difficult math problem or a tricky logic puzzle, the Manager often makes a bad choice.
- The researchers found that inside the frozen team, there were other groups of 8 experts that could have solved the problem much better.
- However, the Manager never picked them. It stuck with its original, weaker choice.
- The Analogy: Imagine a difficult math problem. The Manager sends it to a group of poets and artists. Meanwhile, sitting right next to them in the room, a group of brilliant mathematicians is waiting, ready to solve it perfectly. The Manager just didn't look at them.

Why Does This Happen? (The "Blind Spot")

The paper explains that this isn't because the AI is "stupid." It's because of how the AI was trained.

The Training Flaw: During training, the AI only gets a grade (a score) based on the experts it actually used.
- If the Manager picks the Poets and they do a bad job, the AI gets a bad grade.
- But the AI never sees the grade for the Mathematicians it didn't pick. It doesn't know that the Mathematicians would have gotten an A+.
The Result: The Manager learns to be consistent, but it doesn't learn to be optimal on the hardest problems. It has a "blind spot" for the best possible alternative teams. It's like a student who only studies the answers they wrote down, never checking if a different method would have been faster or better.

The Fix: A Tiny Nudge

The researchers wanted to know: "Is the AI's failure because it doesn't have the skills (capacity), or because the Manager is just picking the wrong people (routing)?"

They tried a very small experiment:

They kept the entire team of 100 experts frozen (no new skills learned).
They kept every other layer of the AI frozen.
They only updated the "Manager" (the router) for the very last layer of the AI.

The Outcome:
Even with this tiny change (updating less than 0.001% of the model's brain), the AI got significantly better at solving hard math and logic puzzles.

The Takeaway: This proves that the AI already had the right experts inside it to solve the hard problems. The failure wasn't a lack of intelligence; it was a routing failure. The Manager just needed a little nudge to pick the right team.

Summary

The Setup: AI models use a "Manager" to pick a small team of experts for each word.
The Problem: On easy words, the Manager is perfect. On hard words, the Manager often picks a weak team when a strong team is sitting right there, unused.
The Cause: The training process only rewards the team that was picked, ignoring the fact that other teams might have been better.
The Proof: By just tweaking the Manager's decision-making rules (without teaching the experts anything new), the AI's performance on hard tasks improved. This means the "experts" were ready; the "Manager" just needed to wake them up.

In short: The AI isn't necessarily failing because it lacks knowledge; it's failing because the person assigning the tasks isn't assigning them to the right people when things get difficult.

Technical Summary: When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

1. Problem Statement

Mixture-of-Experts (MoE) language models scale capacity by routing each token to a small subset of experts ( $k$ ) out of a large pool ( $N$ ). While the router is typically trained via standard language modeling loss and auxiliary load-balancing objectives, the quality of the specific routing decisions made by a trained top- $k$ router has rarely been evaluated directly.

The central question addressed is: Are the routes selected by a standard trained router actually the best ones available within the frozen model?

Prior work assumes that routing stabilizes early and that downstream performance is sufficient evidence of good routing. However, reasoning success often depends disproportionately on a small subset of "pivotal" or high-entropy tokens. The authors hypothesize that standard training may fail to optimize routing for these critical, low-confidence tokens because the training signal does not evaluate alternative, equal-compute routes.

2. Methodology

Counterfactual Routing Analysis

The authors propose a protocol to evaluate routing quality without modifying the model's weights (except for a specific intervention experiment).

Setup: Using verified, correct reasoning trajectories (e.g., from MATH Level-5, AIME, HMMT), the authors analyze tokens at specific MoE layers.
Sampling Alternatives: For a given token $t$ at layer $\ell$ , instead of only observing the standard top- $k$ route ( $S^{\text{std}}_{t,\ell}$ ), they sample $G=32$ alternative top- $k$ routes from the same pool of high-scoring experts using Gumbel-top- $k$ sampling. All alternatives have the same computational cost (same $k$ ).
Scoring: For each route (standard and alternatives), the model is run forward with that specific route active at layer $\ell$ (all other parameters frozen). The utility of a route is scored by the probability it assigns to the realized next token in the verified trajectory.
Stratification: Tokens are grouped into three bins based on the mean probability assigned by the sampled alternative routes ( $\bar{p}$ $\overset{p}{ˉ}$ ):
- Confident: $\bar{p} > 0.9$
- Ambiguous: $0.5 < \bar{p} \le 0.9$
- Fragile: $\bar{p} \le 0.5$

Structural Analysis of Training

The paper analyzes the gradient flow in standard top- $k$ MoE training to explain observed failures. It demonstrates that the language modeling loss only provides a signal for the executed route. The gradient for unselected experts vanishes. Furthermore, standard load-balancing losses (e.g., Switch-style) depend only on aggregate statistics (router scores and selection frequencies) and do not provide token-level counterfactual signals regarding which alternative route would have minimized loss.

Minimal Router Update (Expert Preference Optimization - EPO)

To test if routing failures are "reachable" within the frozen model, the authors perform a minimal intervention:

Freeze: All expert weights and all router weights except the final layer are frozen.
Train: Only the final-layer router is updated using Expert Preference Optimization (EPO).
Mechanism: EPO identifies "hard tokens" (high cross-entropy). It samples alternative routes; if an alternative yields a lower cross-entropy than the current selection, it is treated as the "chosen" route ( $r^+$ ) and the current one as "rejected" ( $r^-$ ). The router is updated via a DPO-style objective to prefer $r^+$ over $r^-$ .
Evaluation: The updated router is tested on downstream benchmarks (AIME, HMMT) without test-time search.

3. Key Results

Routing Quality Degrades on Fragile Tokens

Across four models (Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, OLMoE-1B-7B) and multiple benchmarks:

Confident Tokens: The standard router is well-aligned with utility. The standard route is often the best among sampled alternatives, and the probability gap is negligible.
Fragile Tokens: The alignment collapses.
- The standard route is the best among sampled alternatives in only 0.8% of fragile tokens (for Qwen3).
- The best sampled equal-compute route improves the next-token probability by 20.4 percentage points on average for Qwen3 on fragile tokens.
- This pattern holds across layers (though the gap widens at deeper layers) and across different model families.

The Counterfactual Blind Spot

The analysis confirms that standard training creates a "counterfactual blind spot."

The loss function only evaluates the executed route.
Load balancing encourages aggregate diversity but does not signal which specific alternative route is better for a specific hard token.
Consequently, the router learns to route confidently well but fails to distinguish between suboptimal and optimal routes on difficult tokens where the choice matters most.

Minimal Update Shifts Performance

The EPO experiment demonstrates that the failure is not solely due to a lack of expert capacity:

Updating less than 0.001% of the model parameters (only the final-layer router) was sufficient to shift pass@K performance on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B.
The improvement is concentrated on Ambiguous and Fragile tokens, where the updated router selects better routes, while performance on Confident tokens remains stable (though rank alignment metrics shift slightly due to reduced bias toward easy tokens).

4. Significance and Claims

The paper argues that routing quality should be treated as a first-class concern in MoE training, rather than a mere implementation detail of sparsity.

Diagnosis: The standard router is not uniformly "bad"; it is highly effective on confident tokens but systematically misaligned on the fragile tokens that drive hard reasoning.
Root Cause: This misalignment is a structural consequence of standard top- $k$ training objectives, which lack token-level counterfactual signals for unselected routes.
Implication: High loss on hard tokens is not necessarily a capacity limit of the experts; it can be a routing failure that is reachable within the frozen model.
Future Direction: The authors suggest that future pretraining objectives should incorporate token-level signals on equal-compute alternative routes (e.g., evaluating sampled alternatives alongside the executed one) to prevent this misalignment during training, rather than relying on post-hoc fixes.

The authors remain modest, noting that their analysis is performed on verified-correct trajectories (where the realized token is known to be correct) and that they analyze layers independently. However, the consistency of the pattern across diverse models and domains suggests a fundamental limitation in current MoE training paradigms.