💬 NLP

Mixture-of-Experts as Soft Clustering: A Dual Jacobian-PCA Spectral Geometry Perspective

This paper introduces a Dual Jacobian-PCA spectral analysis to demonstrate that Mixture-of-Experts architectures function as soft clusterings that reduce local function curvature and increase representation effective rank by decomposing the model into low-overlap, expert-specific transformations.

Original authors: Feilong Liu

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Feilong Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: What is a Mixture-of-Experts (MoE)?

Imagine you are running a massive, high-tech call center.

The Old Way (Dense Model): Every single call goes to every agent in the building. Everyone listens to every question, even if they are an expert in plumbing and the caller is asking about coding. It's chaotic, loud, and inefficient.
The New Way (MoE): You have a smart receptionist (the Router). When a call comes in, the receptionist listens to the problem and immediately sends it to the one or two agents who are actually experts at that specific topic. The rest of the call center stays quiet.

This paper asks a fascinating question: Does this "smart routing" actually change how the agents think and learn, or does it just save money?

The authors say: Yes, it changes their thinking. It makes them "flatter," more specialized, and less sensitive to tiny changes.

The Two Main Tools: The "Microscope" and the "Map"

To understand this, the researchers used two special tools to look inside the AI's brain:

1. The "Sensitivity Microscope" (Jacobian Analysis)

Imagine an agent is holding a very sensitive seismograph (a machine that measures earthquakes).

The Dense Agent: If you tap the table slightly, the seismograph goes crazy. The agent is hyper-sensitive. A tiny change in the input causes a huge, chaotic reaction in the output.
The MoE Agent: The same tiny tap barely moves the needle. The agent is calm and stable.
The Finding: The paper found that MoE agents are much "flatter." They don't overreact to small mistakes or noise. This is good because it means the AI is less likely to hallucinate (make up wild answers) when the input is slightly imperfect.

2. The "Variance Map" (Weighted PCA)

Imagine the agents are trying to organize a messy pile of LEGO bricks (data).

The Dense Agent: They try to sort the whole pile at once. They end up with a few huge, messy piles where everything is mixed together. They rely on just a few big categories to make sense of the chaos.
The MoE Agent: Because the receptionist sent them only specific bricks (e.g., "only red 2x4s"), they can organize their specific pile with incredible detail. They use many different, smaller categories to sort the bricks.
The Finding: MoE agents have a higher "effective rank." They don't just squish everything into a few big buckets; they spread the information out across many nuanced directions. They see more detail in their specific niche.

The Twist: Hard vs. Soft Routing

The paper also tested two different types of receptionists:

The Strict Receptionist (Top-k Routing): "You are a plumbing expert. ONLY talk to plumbing calls. Ignore everything else."
- Result: The experts become very focused. Their internal organization is tight and concentrated. They are like specialists in a narrow field.
The Chatty Receptionist (Fully-Soft Routing): "You are a plumbing expert, but you can also listen to some electrical calls if you have time."
- Result: The experts are more relaxed. They see a wider variety of inputs, so their internal organization is broader and less concentrated.

The Surprise: Even though the "Chatty" receptionist lets experts hear more, the experts still don't overlap much with each other. They still develop their own unique ways of thinking. They don't just copy each other.

The "Real World" Test: Synthetic vs. Language

The researchers first tested this with random noise (like static on a TV).

Result: The MoE experts were like specialists sorting specific colors of noise.

Then, they tested it on real text (like a Wikipedia article).

Result: This is where it got weird. With real language, the "Dense" model (the one trying to do everything at once) actually became more complex and messy because language is so structured and hierarchical.
The MoE experts, however, acted like local guides. They realized, "Oh, this paragraph is about history, so I'll use my history map," while another expert said, "This paragraph is about math, so I'll use my math map."
The Takeaway: MoEs are like soft-clustering. They break the giant, confusing world of language into smaller, manageable "neighborhoods" (charts) that each expert knows intimately.

Why Should You Care? (The Three Big Takeaways)

The authors suggest three exciting possibilities for the future of AI based on this geometry:

The "Goldilocks" Number of Experts: There is likely a perfect number of experts (an "elbow point") where adding more doesn't help because the "neighborhoods" are already perfectly sorted.
Less Hallucination: Because MoE experts are "flatter" (less sensitive), they might be less likely to go off the rails and make up fake facts when the conversation gets tricky.
Better Teamwork: The "Strict" (Top-k) routing creates a team where everyone has a totally different perspective (low overlap). This makes the whole team more reliable, like a jury of diverse experts rather than a group of clones.

Summary in One Sentence

This paper proves that Mixture-of-Experts models aren't just a way to save computing power; they fundamentally change the AI's brain into a team of calm, specialized experts who organize information differently than a single, overworked generalist, making them more stable and less prone to errors.

1. Problem Statement

While Mixture-of-Experts (MoE) architectures are widely adopted in Large Language Models (LLMs) for their efficiency and conditional computation capabilities, the geometric impact of expert routing on learned functions and internal representations remains poorly understood.

The Gap: Prior research focuses on scaling laws, routing strategies, and downstream performance. However, it is unclear how routing mechanisms (e.g., Top-k vs. fully-soft) reshape the spectral geometry of the model, specifically regarding local function sensitivity (curvature), cross-expert alignment, and the dimensionality of expert-local representations.
The Challenge: Analyzing these geometric properties in large-scale, attention-based models is difficult due to computational constraints and the complexity of isolating routing effects from other architectural factors.

2. Methodology

The authors propose a "Dual Jacobian-PCA Spectral Geometry Probe" to analyze MoE architectures in a controlled setting that permits exact mathematical analysis.

A. Experimental Setup

Architecture: A simplified MLP-based MoE architecture (omitting attention and layer normalization) to isolate routing effects.
Data: Synthetic isotropic Gaussian inputs (for stable covariance estimation) and WikiText-2 (for validation on natural language).
Comparisons: Dense baselines vs. MoE models with matched capacity, comparing Top-k routing (sparse) and fully-soft routing (dense weights).

B. Geometric Probes

Jacobian Spectral Probe (Function Geometry):
- Computes the singular-value spectrum of the expert-local Jacobian ( $J_e(x) = \partial f_e(x) / \partial x$ ).
- Metrics: Leading singular values (local sensitivity/norm), spectral decay rate, and cumulative spectral energy.
- Cross-Expert Alignment: Computes cosine similarity between flattened average Jacobians of different experts to measure functional overlap.
Weighted PCA Probe (Representation Geometry):
- Performs Principal Component Analysis on hidden states entering the MoE layer.
- Weighting: Uses routing weights ( $g_e(x)$ ) as sample weights to ensure the PCA reflects the specific input distribution each expert actually processes.
- Metrics: Explained variance ratios, cumulative variance curves, and effective rank (number of components needed to reach a variance threshold).

3. Key Contributions

Geometric Framing: Interprets MoEs as inducing soft partitionings of function space into overlapping "expert-local charts," providing a mechanistic view of specialization.
Spectral Comparison: Systematically demonstrates how routing sharpness (Top-k vs. Soft) modulates local sensitivity and representation variance under matched capacity.
Novel Probes: Introduces a combined framework using Jacobian spectra and weighted PCA to quantify expert-local structure, revealing that routing decomposes sensitivity into low-overlap components while redistributing variance.

4. Key Results

A. Synthetic Data (MLP-MoE) Findings

Reduced Local Sensitivity: MoE experts exhibit smaller leading singular values (1.5–1.7× lower than dense baselines) and faster spectral decay. This indicates that routing flattens local curvature and reduces worst-case sensitivity to input perturbations.
Low Cross-Expert Alignment: Pairwise cosine similarity between expert Jacobians is near zero ( $\approx 0$ ). Experts learn distinct transformations with limited overlap rather than scaled variants of a shared map.
Redistributed Variance (Higher Effective Rank): Under weighted PCA, MoE experts show slower eigenvalue decay and require more principal components to reach 90% cumulative variance compared to dense models. This suggests MoEs redistribute variance across more directions, increasing effective rank.
Routing Sharpness Effects:
- Top-k Routing: Yields more concentrated variance (lower effective rank) and sharper spectral decay. Experts operate on narrower input regions.
- Fully-Soft Routing: Produces broader variance structure and flatter PCA spectra, exposing experts to more heterogeneous inputs.

B. Natural Language Validation (WikiText-2 Transformer)

Curvature Reduction Persists: The reduction in Jacobian norms (flatter mappings) holds true in a 3-layer transformer, confirming the synthetic findings.
Inversion of PCA Results: Unlike the synthetic case, on natural language data, MoE experts show lower effective rank (fewer components needed for 90% variance) than the dense baseline.
- Interpretation: Natural language data lies on a low-dimensional, anisotropic manifold. Dense layers must mix heterogeneous contexts (inflating rank), while MoE routing soft-clusters tokens into simpler, locally lower-dimensional sub-manifolds, allowing experts to compress representations more effectively.

5. Significance and Implications

The paper provides a mechanistic baseline for understanding MoE behavior, moving beyond efficiency metrics to geometric insights.

Geometric Elbow Hypothesis: Suggests an optimal expert count exists based on variance concentration, similar to clustering "elbows."
Hallucination Suppression: The observed flattening of local mappings (lower Jacobian norms) implies MoEs may be less sensitive to infinitesimal perturbations, potentially reducing error accumulation and hallucinations in long-context generation.
Ensemble Diversity: Top-k routing creates strictly lower cross-expert alignment than soft routing, suggesting superior ensemble diversity and functional orthogonality.
Design Criteria: Offers actionable guidelines for balancing routing sharpness and representation rank in frontier models. The authors propose that MoEs effectively flatten function space (reducing sensitivity) while compressing representation space (on structured data like text) by partitioning the manifold.

Conclusion

This work establishes that MoE routing is not merely a computational optimization but a geometric mechanism that decomposes function space into low-overlap, specialized charts. By flattening local curvature and adapting representation dimensionality to the data manifold, MoEs achieve a unique balance of stability and expressivity that dense architectures cannot replicate under the same capacity constraints.