💬 NLP

Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

This paper identifies and addresses the "Seeing but Not Thinking" phenomenon in Multimodal Mixture-of-Experts models, where visual inputs cause routing mechanisms to under-activate task-relevant reasoning experts, and proposes a routing-guided intervention that significantly improves complex visual reasoning performance by ensuring proper expert activation.

Original authors: Haolei Xu, Haiwen Hong, Hongxing Li, Rui Zhou, Yang Zhang, Longtao Huang, Hui Xue, Yongliang Shen, Weiming Lu, Yueting Zhuang

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haolei Xu, Haiwen Hong, Hongxing Li, Rui Zhou, Yang Zhang, Longtao Huang, Hui Xue, Yongliang Shen, Weiming Lu, Yueting Zhuang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Eyes-Open, Brain-Asleep" Phenomenon

Imagine you have a super-smart student named MoE (Mixture of Experts). This student has a library of 100 different tutors inside their head, but for any given question, they only wake up a few of them to help.

The researchers noticed something weird happening with MoE:

Scenario A (Text): You give MoE a math word problem written on a piece of paper. MoE reads it, thinks hard, and gets the answer right.
Scenario B (Image): You take that exact same math problem, write it on a whiteboard, take a photo, and show MoE the picture. MoE looks at the photo, correctly reads every number and letter (perfect vision!), but then gives the wrong answer.

The researchers call this "Seeing but Not Thinking." The model's eyes are wide open, but its reasoning brain is asleep.

The Investigation: Why is this happening?

The team asked: Is the model bad at reading the picture?
No. They checked, and the model could read the text in the image perfectly.

Then they asked: Is the model confused because "pictures" and "words" feel different to it?
No. They found that deep inside the model, pictures and words actually share the same "language" in the middle layers of the brain. The model understands the meaning of the image just fine.

The Real Culprit: The "Distracted Traffic Cop"

So, if the model can see and understand, why does it fail? The paper introduces a new idea called Routing Distraction.

Think of the model's brain as a busy office building with many specialized departments (Experts):

Visual Department: Good at recognizing shapes and colors.
Math Department: Good at solving logic puzzles.
Language Department: Good at grammar.

In a normal office, a manager (the Router) looks at a task and calls the right department.

Text Task: The manager sees "Math Problem" and calls the Math Department.
Image Task: The manager sees "Math Problem in a Picture."

Here is the glitch: When the task comes in as an image, the manager gets distracted by the "Visual Department." Even though the problem is a math puzzle, the manager sends the file to the Visual Department first, or splits the team up in a way that leaves the Math Department understaffed.

The model is essentially looking at the picture so hard that it forgets to do the math. The "Visual" experts take up the spotlight in the middle of the process, pushing the "Reasoning" experts out of the way.

The Solution: The "Nudge"

To fix this, the researchers invented a Routing-Guided Intervention.

Imagine the manager (Router) is about to send the file to the Visual Department. The researchers step in and gently nudge the manager's hand, saying, "Hey, remember, this is a math problem! Make sure the Math Department gets a bigger share of the work."

They didn't retrain the whole model (which is expensive and slow). They just adjusted the "traffic signals" during the thinking process to ensure the Math Experts got activated, even when looking at a picture.

The Results

When they applied this "nudge" to three different AI models:

It worked. The models started solving image-based math problems much better.
It was smart. The "nudge" didn't just force the model to guess; it helped the model use the right tools for the job.
It generalized. Even when the pictures were complex (like graphs or geometric shapes), helping the "Math Department" wake up improved the results.

The Takeaway

The paper teaches us that modern AI isn't necessarily "dumb" when looking at images. It's just distracted. It gets so focused on processing the visual details that it forgets to switch into "reasoning mode." By manually guiding the AI to wake up the right "experts" inside its brain, we can help it think clearly again, even when looking at a picture.

In short: The AI isn't blind; it's just looking at the wrong department for the job. We just had to point it in the right direction.

1. Problem Statement: "Seeing but Not Thinking"

The authors identify a critical failure mode in state-of-the-art Multimodal Mixture-of-Experts (MoE) models (e.g., Qwen3-VL, Kimi-VL, Llama4).

The Phenomenon: Models can accurately perceive and extract visual content (e.g., numbers, text in an image) but fail to perform the subsequent logical reasoning required to solve the problem.
The Paradox: When the exact same problem is presented as pure text, the model solves it correctly. However, when presented as an image (even a high-resolution, perfectly legible rendering of the text), the model fails due to reasoning errors, not perception errors.
Quantification: Experiments on the MATH500 dataset show that 68.2% to 73.1% of failures in image-based reasoning stem from reasoning errors, while only ~30% are due to perception errors.

2. Methodology & Analysis

The authors employ a systematic mechanistic analysis to diagnose the root cause, moving beyond the assumption of semantic misalignment.

A. Ruling Out Semantic Alignment Failure

Hypothesis: The visual input fails to align with the textual semantic space.
Experiment: Cross-modal concept intervention. The authors replaced the hidden state of a visual token with the hidden state of a corresponding text token (or vice versa) across different layers.
Finding: An "inverted U-shaped" success rate was observed. Interventions were highly successful (90%+) in middle layers, confirming that cross-modal semantic sharing does exist in MoE architectures. Thus, semantic misalignment is not the primary cause of the reasoning failure.

B. Identifying Layer-wise Expert Separation

Expert Specialization: Using Gini coefficients, the authors found that expert specialization intensifies in deeper layers.
Spatial Distribution:
- Visual Experts: Cluster in early layers (for initial encoding) and terminal layers (for output preparation).
- Domain Experts (Reasoning): Cluster in middle layers.
Key Insight: There is a structural separation where the experts responsible for visual processing and those responsible for domain-specific reasoning occupy different layers.

C. The Routing Distraction Hypothesis

Observation: When processing images, the routing mechanism in the middle layers (where domain experts reside) exhibits significant divergence compared to text inputs.
Correlation: Higher routing divergence (measured by Jensen-Shannon Divergence) in middle layers correlates directly with lower reasoning accuracy.
Hypothesis: The visual modality causes the router to "distract" the computation away from the task-relevant domain experts in the middle layers, directing it toward less suitable experts instead. The model has the reasoning capability but fails to activate the correct computational units.

3. Proposed Solution: Routing-Guided Intervention

To validate the hypothesis and mitigate the issue, the authors propose a test-time intervention strategy.

Domain Expert Identification:
- Compare activation frequencies of experts on domain-specific data (e.g., math problems) vs. general data (e.g., Alpaca).
- Experts with significantly higher activation on domain data are identified as "Domain Experts."
- Robustness: This identification works even if the text reference is not a perfect semantic match to the visual task, as long as it elicits the target reasoning patterns.
Routing Adjustment:
- Soft Intervention: Add a weighted bias to the routing logits of identified domain experts during inference: $r'_{l,k} \leftarrow r_{l,k} + \lambda \cdot s(r_l)$ .
- Hard Intervention: Force domain expert logits to the layer maximum (less effective for some architectures).
- Target Layers: Apply intervention specifically to the middle layers where domain experts are concentrated.

4. Key Results

The method was evaluated on three MoE models (Qwen3-VL, Kimi-VL, Llama4) across six benchmarks, including semantically equivalent image-text pairs and natural visual scenarios.

Performance Gains:
- Semantically Equivalent Scenarios: Consistent improvements across all models. For example, Kimi-VL improved by +2.24% on math and +2.98% on physics.
- Natural Visual Scenarios: The method generalized to complex visual tasks (e.g., MathVerse, MATH-Vision). Kimi-VL achieved a +3.17% gain on MathVerse.
- Overall: Gains of up to 3.17% were observed on complex visual reasoning tasks.
Ablation Studies:
- Layer Specificity: Intervening only in middle layers yielded the best results. Adding early layers often degraded performance (disrupting visual feature extraction).
- Expert Specificity: Using mismatched domain references (e.g., using elementary arithmetic text to guide geometric reasoning) resulted in negligible or negative gains, confirming the need for domain-matched identification.
- Soft vs. Hard: Soft intervention was more robust; hard intervention sometimes caused severe degradation, particularly in models with Top-1 routing mechanisms.

5. Significance and Contributions

Mechanistic Insight: The paper challenges the prevailing view that visual reasoning failures are due to poor perception or semantic misalignment. Instead, it identifies routing distraction as the primary bottleneck in MoE models.
Structural Discovery: It reveals a critical architectural trait: the layer-wise separation between visual processing experts and domain reasoning experts, and how visual inputs disrupt the routing to the latter.
Practical Mitigation: It introduces a lightweight, inference-time intervention method that does not require retraining. By simply adjusting routing weights, models can be guided to utilize their latent reasoning capabilities more effectively.
Generalizability: The approach works across different model scales and architectures, suggesting that "cognitive functions" in MoE models are localized to specific expert clusters that can be targeted for improvement.

Conclusion

The paper concludes that multimodal MoE models possess the necessary reasoning capabilities but suffer from a "routing distraction" where visual inputs prevent the activation of the correct reasoning experts. By explicitly guiding the router to prioritize domain experts in middle layers, this "Seeing but Not Thinking" phenomenon can be effectively mitigated.