🤖 machine learning

Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap

This paper introduces a unified Jacobian-PCA-Grassmann framework to reveal that Mixture-of-Experts (MoE) architectures achieve specialization through a geometric asymmetry where experts exhibit strong functional decorrelation despite operating on partially overlapping representation subspaces, a structure significantly shaped by routing sparsity.

Original authors: Feilong Liu

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Feilong Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Team of Specialists

Imagine a large company (a Mixture-of-Experts or MoE model) that has to solve many different problems. Instead of having one giant employee who tries to do everything, the company hires a team of specialists (the Experts).

When a new task comes in, a "manager" (the Router) looks at the task and decides which specialist is best suited to handle it. The goal is to make the company huge and powerful without making it slow or expensive.

But here's the mystery the paper solves: How do these specialists actually work together? Do they all do the same thing? Do they work in completely separate rooms? Or do they overlap?

The authors built a special "geometric microscope" to look inside these AI models and found a surprising pattern.

The Two Ways to Look at the Experts

To understand the specialists, the researchers looked at them in two different ways:

The "How" (Function Space): They looked at what the experts actually do to the data. Imagine asking, "If I give you a slightly different input, how does your output change?" This is like checking an expert's fingerprint or their unique way of thinking.
The "Where" (Representation Space): They looked at where the experts store their knowledge. Imagine the experts are working in a giant library. Do they all grab books from the exact same shelf, or do they have their own distinct sections?

The Big Discovery: The "Asymmetry"

The paper found a consistent pattern across different AI models (like Mistral and Qwen). It's a bit like a paradox:

They think differently (Functional Decorrelation): When you look at how the experts process information, they are completely different from each other. Their "fingerprints" don't match at all. If Expert A and Expert B were both given the same task, they would solve it using totally different mental steps. They are not redundant; they are truly unique.
But they share the same room (Representational Overlap): However, when you look at where they store their results, they aren't in totally separate rooms. They are working in the same library, and their "shelves" (subspaces) overlap significantly. They are looking at the same books, just interpreting them differently.

The Analogy:
Imagine a group of chefs in a kitchen.

The "How": If you ask them to chop an onion, Chef A uses a specific rocking motion, while Chef B uses a straight-up-and-down motion. Their techniques are totally different (Functional Decorrelation).
The "Where": But, they are both standing at the same cutting board, using the same knife, and handling the same pile of onions. They aren't in separate kitchens; they are sharing the same workspace (Representational Overlap).

The Secret Ingredient: How the Manager Decides

The researchers also tested what happens if they change the "manager's" rules for picking chefs.

Strict Manager (Top-k Routing): The manager picks only the one best chef for the job.
- Result: The chefs become very distinct. They sharpen their unique skills, and their workspaces become more separate. The "fingerprint" difference gets even clearer.
Lenient Manager (Fully-Soft Routing): The manager lets all chefs work on the task at the same time, mixing their efforts.
- Result: The chefs start to blur together. They lose their unique styles, and their workspaces become a messy, tangled overlap. They start doing the same thing.

The Takeaway: The "strictness" of the manager is what keeps the specialists distinct. If you let everyone do everything, they stop being specialists and start being clones.

Why This Matters (According to the Paper)

This discovery changes how we understand these AI models:

They aren't just copying each other: Even though they share the same "workspace," they are doing genuinely different calculations.
They aren't in separate worlds: They aren't partitioning the world into "my job" and "your job." Instead, they are all looking at the same complex reality but applying different, unique transformations to it.
The "Soft Partition": The paper suggests these models work like a soft partition. Imagine a Venn diagram where the circles overlap. The experts operate in the overlapping areas, but they apply their own unique "flavor" to the data.

Summary

The paper introduces a new way to measure AI experts. It found that in modern AI models, experts are functionally unique (they think differently) but spatially shared (they work in the same space). The "sharpness" of the routing mechanism (how strictly the model picks experts) is the dial that controls how distinct these experts remain. If the routing is too loose, the experts lose their individuality.

Technical Summary: Geometric Asymmetry in MoE Specialization

Problem Statement

While Mixture-of-Experts (MoE) architectures have become a standard mechanism for scaling Transformer models by enabling large parameter counts with fixed computational costs, the internal geometric structure of expert specialization remains poorly understood. Existing analyses largely focus on routing statistics, load balancing, or system-level performance. A critical gap exists in understanding how experts differ in function space (the transformations they perform) versus representation space (how routed tokens are organized in hidden states).

Current intuition suggests MoE layers partition the input distribution into disjoint clusters where each expert acts as a local approximator. However, empirical evidence suggests experts often operate on overlapping token regimes, raising the question: What geometric structure does routing induce inside a pretrained MoE Transformer? Specifically, do experts implement distinct functions, and how do their representations relate to one another?

Methodology

The authors introduce a unified Jacobian-PCA-Grassmann framework to analyze MoE layers in both function and representation spaces. This framework is applied to pretrained models (Mistral, Qwen) and controlled 3-layer Transformer models trained from scratch.

1. Functional Geometry: Expert-Local Jacobians

To characterize the function of individual experts independent of routing dynamics, the authors analyze the expert-local Jacobian ( $J_e(x) = \partial f_e(x) / \partial x$ ).

Metric: They compute the average Jacobian ( $\bar{J}_e$ ) for each expert over a dataset and vectorize it.
Alignment: Pairwise functional alignment is measured via the normalized Frobenius inner product (cosine similarity) between these vectorized Jacobians.
Interpretation: Low cosine similarity indicates functional decorrelation, meaning experts implement locally distinct transformations rather than redundant variants of the same function.

2. Representation Geometry: Routed PCA and Grassmannian Distances

To characterize the geometry of routed representations, the authors collect hidden states weighted by routing weights ( $h_e(x) = g_e(x)h(x)$ ).

PCA: Standard Principal Component Analysis (PCA) is performed on the routed hidden states to identify principal directions and variance spectra.
Subspace Distance: The geometric separation between experts is quantified using the Grassmannian Geodesic Distance between the top- $n$ PCA subspaces of different experts.
Interpretation: This metric captures multi-directional separation between entire subspaces. A distance near zero implies redundant functional roles, while a distance approaching the theoretical maximum ( $\frac{\pi}{2}\sqrt{n}$ ) implies orthogonality.

3. Controlled Routing Experiments

To isolate the effect of routing mechanisms from training dynamics, the authors compare two regimes on a controlled 3-layer model with fixed expert parameters:

Top-k Routing: Sparse, discrete selection.
Fully-Soft Routing: Dense, continuous mixture.

Key Results

1. Structural Asymmetry: Functional Decorrelation vs. Representation Overlap

Across pretrained MoE Transformers (Mistral-8x7B and Qwen1.5-MoE-A2.7B), the authors observe a consistent structural asymmetry:

Strong Functional Decorrelation: Cross-expert Jacobian alignment is consistently near zero (mean $\approx$ 0.000–0.068). This indicates that experts are not redundant; they implement distinct local transformations.
Partial Representation Overlap: Despite functional decorrelation, the routed representations occupy distinct but partially overlapping subspaces. Grassmannian distances are substantial (indicating separation) but remain significantly below the theoretical maximum for complete orthogonality.
Conclusion: Functional decorrelation and representation overlap coexist. Experts act as locally decorrelated operators over overlapping regimes on a shared manifold, rather than partitioning the input space into disjoint clusters.

2. Routing Sparsity as a Geometric Control Knob

Controlled experiments reveal that routing sharpness directly shapes this geometry:

Top-k Routing: Induces sharper functional separation (low Jacobian alignment) and larger Grassmannian distances (greater subspace divergence).
Fully-Soft Routing: Yields entangled expert structures with high Jacobian alignment (indicating collapse toward redundant mappings) and collapsed Grassmannian distances (high subspace overlap).
Implication: Routing sparsity is a key factor in enforcing specialization.

3. Comparison with Dense Layers

MoE experts exhibit steeper eigenvalue decay in their variance spectra compared to dense feedforward layers. The first few principal components capture a significantly higher percentage of variance in MoE experts, suggesting that routing concentrates tokens into lower-dimensional, more focused subspaces.

4. Cross-Model Consistency

The observed geometric patterns (low Jacobian alignment + high but sub-maximal Grassmann separation) are consistent across different models (Mistral, Qwen), layers (early, middle, late), and datasets, suggesting this is a robust property of MoE architectures rather than an artifact of specific training runs.

Significance and Claims

The paper claims to provide a general diagnostic framework for studying conditional computation in modern Transformer architectures. Its primary contributions are:

Unified Diagnostic Tool: It offers a systematic method to analyze expert specialization beyond simple routing statistics or activation patterns, jointly characterizing function space and representation space.
Geometric Interpretation of MoE: The results support a view of MoE layers as implementing a soft partitioning of a shared representation manifold. Experts are functionally distinct but operate on overlapping submanifolds.
Mechanistic Insight: The study identifies routing sharpness (sparsity) as a direct control mechanism for the degree of specialization, linking architectural design choices to measurable geometric structures.
Implications for Model Design: The authors suggest that strategies for expert merging or pruning based solely on representation similarity (e.g., activation overlap) may be flawed. Because experts can have overlapping representations while performing distinct functions, functional metrics (like Jacobian alignment) may be required to correctly identify true redundancy.

The paper maintains a modest scope, acknowledging that its analysis is local (first-order Jacobians) and linear (PCA), and does not capture global nonlinear behavior or the evolution of geometry during training. However, it establishes a foundational geometric perspective on how MoE models organize computation.