🤖 machine learning

Eigenvectors of Experts are Training-free Non-collapsing Routers

This paper introduces SSMoE, a training-free framework that leverages the eigenvectors of expert weight matrices via Singular Value Decomposition to effectively resolve the expert collapse issue in Sparse Mixture of Experts models and enhance their performance across diverse tasks.

Original authors: Giang Do, Hung Le, Truyen Tran

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Giang Do, Hung Le, Truyen Tran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, highly specialized library (a Large Language Model) designed to answer any question you throw at it. Inside this library, there are hundreds of expert librarians (called "Experts"). When you ask a question, a "Head Librarian" (called the "Router") decides which few experts should help you.

The Problem: The "Groupthink" Glitch
The paper identifies a major flaw in how these libraries currently work. Even though the Head Librarian is supposed to pick the best experts for each specific question, they often get stuck in a rut. They keep picking the same few experts over and over, ignoring the others.

The authors call this "Expert Collapse." It's like a restaurant where the manager keeps sending every customer to the same two chefs, even though the kitchen has 50 other talented chefs. The result? The restaurant becomes inefficient, the food quality drops, and the unused chefs (the other experts) just sit there doing nothing.

Previous attempts to fix this involved retraining the Head Librarian from scratch, which is like hiring a new manager and spending months teaching them how to run the kitchen. This is expensive and slow.

The Discovery: The Experts' ID Cards
The researchers asked a simple question: Can we route questions without a Head Librarian at all?

They looked inside the "ID cards" (the mathematical weight matrices) of the experts themselves. They discovered something surprising: The experts' own internal structures already contain a map of what they are good at. Specifically, the "eigenvectors" (a fancy math term for the main directions of information) in these ID cards hold rich, semantic meaning.

Think of it this way: Instead of asking a manager who is best at cooking Italian food, you look at the chef's own resume and see that their name is literally written in "Italian." The experts are already shouting, "I know math!" or "I know poetry!" through their own internal structure.

The Solution: SSMoE (The Self-Organizing Library)
The paper proposes a new system called SSMoE. Instead of using a learned Head Librarian to guess who to pick, SSMoE uses the experts' own "ID cards" (their eigenvectors) to route the questions.

No Training Needed: This is the biggest win. You don't need to retrain the model or spend money on new data. You just look at the existing math inside the model and use it to route tokens. It's "training-free."
No Collapse: Because the experts' internal maps are naturally different from each other (mathematically "orthogonal"), they don't collapse into a groupthink. They naturally spread the work out.
Better Performance: By using this method, the model actually gets smarter. It uses the right experts more often, leading to better answers.

The Results: Smarter, Faster, and Stronger
The authors tested this on various "libraries" (models) ranging from small ones to massive 120-billion-parameter giants.

Smarter Answers: On difficult reasoning tasks (like math and science questions), SSMoE outperformed the original models. For example, on a 20-billion parameter model, it improved accuracy by about 6% on average.
Saves Money (Memory): Because the routing is so efficient, they could actually remove 25% of the experts (fire the ones that weren't needed) and still get better results than the full original model. This saved about 23% of the computer memory needed to run the model.
Robustness: Even when the input data was messy or corrupted (like a question with random typos or noise), SSMoE held up better than the traditional models. It was less likely to get confused.

In a Nutshell
The paper shows that we don't need a complicated, expensive manager to tell experts what to do. If we just listen to the experts' own internal "vibes" (their eigenvectors), they can organize themselves perfectly. This makes AI models more efficient, more accurate, and less prone to the "groupthink" errors that usually plague them, all without needing to retrain them.

Technical Summary: Eigenvectors of Experts are Training-free Non-collapsing Routers

Problem Statement

Sparse Mixture of Experts (SMoE) architectures have become a cornerstone for scaling Large Language Models (LLMs) and vision-language models by routing input tokens to a subset of specialized experts. However, these models suffer from a persistent issue known as expert collapse (or representation collapse), where distinct experts converge to produce similar or redundant outputs, thereby degrading the model's effective capacity and performance.

While prior research has attempted to mitigate this by improving the router (the mechanism that selects experts), these approaches typically require training from scratch or fine-tuning, incurring high computational and data-processing costs. Furthermore, empirical evidence presented in this work demonstrates that even well-pretrained SMoE models, including state-of-the-art systems like GPT-OSS, Qwen3, and DeepSeek-MoE, continue to exhibit significant router collapse across various layers and model scales. This persistence suggests that conventional router improvements are insufficient and that a fundamental re-examination of the routing mechanism is necessary.

Methodology: Singular Value Decomposition SMoE (SSMoE)

The authors propose SSMoE, a novel, training-free framework that eliminates the reliance on learned routers by leveraging the spectral properties of the expert weight matrices themselves.

Core Insight

Through an analysis of advanced, pretrained SMoE models, the authors observe that the eigenvectors of the expert weight matrices encode rich semantic information. Unlike learned router embeddings, which can collapse, these eigenvectors naturally maintain orthogonality and capture intrinsic expert specialization.

The SSMoE Framework

The method operates in four primary steps:

Eigenvector Extraction: For each expert $i$ , the authors compute two Gram matrices derived from the expert's Feedforward Neural Network (FFN) weights ( $FFN_1$ and $FFN_2$ ). They then extract the eigenvectors of these matrices.
Spectral Embedding Construction: For a given expert, the top- $c$ eigenvectors most similar to the expert's router embedding are selected from both Gram matrices. These are averaged to form a "spectral embedding" ( $C^{(i)}$ ) that represents the expert's semantic specialization.
Routing: The spectral embeddings of all experts are concatenated to form a routing weight matrix ( $W_{EV}$ ). The input token $x$ is projected through this matrix to generate eigenvector-based logits ( $f_{EV}(x)$ ).
Hybrid Routing: To balance the new spectral signal with the original context-aware router, SSMoE employs a convex combination:
$V(x) = \alpha f_{EV}(x) + (1 - \alpha) s(x)$
where $s(x)$ is the original learned router logits and $\alpha \in [0, 1]$ is a tunable balancing factor. Empirical results suggest $\alpha$ values between 0.5 and 0.9 yield optimal performance.

Theoretical Guarantee

The paper provides a theoretical lemma demonstrating that the SSMoE router reduces pairwise logit correlations. Specifically, if the eigenvector logits are orthogonal to the learned router logits, the covariance of the combined router is strictly reduced by a factor of $(1-\alpha)^2$ compared to the original router. This mathematically proves that SSMoE mitigates the collapse problem inherent in conventional SMoE routers.

Key Contributions

Identification of Collapse in Pretrained Models: The authors empirically demonstrate that router collapse is a pervasive issue across ten state-of-the-art MoE-based models (ranging from 4B to 120B parameters), affecting both reasoning and non-reasoning architectures.
Semantic Encoding in Eigenvectors: The work reveals that the eigenvectors of expert weight matrices naturally encode rich semantic information, serving as a robust alternative to learned router embeddings.
SSMoE Framework: The introduction of a training-free, efficient framework that utilizes these spectral properties to enhance token routing and expert representation without requiring additional training.
Expert Pruning Strategy: The proposal of an expert dropping method based on SSMoE that improves performance while significantly reducing memory consumption.

Experimental Results

The authors evaluated SSMoE across diverse tasks and model scales under both clean and corrupted (noisy) data settings.

Large Reasoning Models (GPT-OSS): On GPT-OSS-20B and GPT-OSS-120B, SSMoE outperformed the original models and baseline random dropping strategies.
- It achieved an average improvement of ~6% over original GPT-OSS models on reasoning benchmarks (ARC, GSM8K, etc.) while reducing memory consumption by ~23%.
- In a 10-shot setting, SSMoE improved performance by 21% over baselines for the 20B model and 10% for the 120B model.
- Crucially, ablation studies showed that performance gains were driven by the routing mechanism itself, not merely by reducing the number of experts.
Large Language Models (MTEB): Evaluated on OLMoE, Qwen-MoE, and DeepSeekMoE, SSMoE consistently outperformed baselines (Router, SMoE, MoEE) on the Massive Text Embedding Benchmark (MTEB).
- It achieved state-of-the-art performance on 15/18 tasks in clean settings and 16/18 tasks in corrupted settings.
- The method demonstrated superior robustness under adversarial perturbations (corrupt data), outperforming complex representations that suffered significant performance drops.
Vision-Language Models (CLIP-MoE): On zero-shot image-text retrieval (COCO, Flickr30k) and classification tasks, SSMoE improved Recall@1 by up to +1.1% and demonstrated greater robustness to image noise compared to the baseline CLIP-MoE.
Efficiency: The framework reduced memory usage by approximately 23% in reasoning tasks and 5% in GFLOPs for embedding tasks, while maintaining or improving accuracy.

Significance and Claims

The paper claims that SSMoE represents a paradigm shift in how SMoE architectures can be optimized. By shifting the focus from learning a router to analyzing the intrinsic spectral properties of the experts, the authors demonstrate that:

Training-free optimization is viable: Significant performance gains and robustness can be achieved without the computational cost of retraining or fine-tuning.
Internal structure matters: A deeper understanding of model internals (specifically the orthogonality of eigenvectors) can guide the development of more effective architectures.
Robustness: The eigenvector-based representation provides a more stable and semantically rich signal that is less prone to collapse and more resilient to data corruption than traditional learned routers.

The authors conclude that their findings underscore the importance of leveraging spectral insights to create more efficient, effective, and robust SMoE architectures, offering a practical solution to the long-standing problem of expert collapse.