💬 NLP

Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics

This paper investigates the geometric origins of anisotropy in Transformer models by theoretically demonstrating how frequency-biased sampling and training dynamics shape curvature and tangent directions, and empirically validating through mechanistic interpretability that activation-derived tangent proxies capture significantly more gradient energy and anisotropy than standard controls across various architectures.

Original authors: Raphael Bernas, Fanny Jourdan, Antonin Poché, Céline Hudelot

Published 2026-04-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Raphael Bernas, Fanny Jourdan, Antonin Poché, Céline Hudelot

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Crowded Room" Problem

Imagine a giant, empty ballroom where a Transformer model (a type of AI) is learning to speak. Every word it learns is a person entering the room.

For a long time, researchers noticed something strange: instead of these "word people" spreading out evenly across the whole ballroom, they all huddled together in one tiny, narrow corner. They were so close to each other that even words with totally different meanings (like "apple" and "car") ended up standing shoulder-to-shoulder.

In math terms, this is called anisotropy. It's like a crowded subway car where everyone is squished against the door, making it hard to tell who is who or move around freely. This usually causes problems for the AI when it tries to understand nuances.

The Paper's New Idea:
This paper argues that this "crowding" isn't a mistake or a bug. It's actually a feature of how the AI learns, driven by two main forces: Frequency (how often a word is used) and Geometry (the shape of the learning space).

Analogy 1: The "Frequent Flyer" vs. The "Tourist"

The paper explains that the AI treats common words and rare words very differently.

The Frequent Flyers (Common words like "the," "is," "and"): These words appear millions of times. Because they are seen so often, the AI gets very confident about them. They become "pinned down" to a specific spot. Imagine a famous celebrity who is so recognizable they always stand in the exact same spot in the ballroom. They don't move much; they are rigid and stable.
The Tourists (Rare words like "quintessential" or "flabbergasted"): These words appear rarely. The AI is less sure about them, so they wander around more. They have more "wiggle room" and explore different parts of the room.

The Result: Because the "Frequent Flyers" are so numerous and rigid, they form a dense, narrow cone of people. The "Tourists" get pushed into the same narrow cone just to stay near the crowd. This creates the "anisotropy" (the squished corner).

Analogy 2: The "Trampoline" and the "Tangent"

To understand why this happens, the authors use some fancy geometry, but we can simplify it with a trampoline.

Imagine the AI's knowledge is a giant, bouncy trampoline (the Manifold).

The Tangent (The Flat Part): If you stand on a trampoline, the surface right under your feet is flat. This is the "tangent."
The Curvature (The Dip): If you jump, the trampoline curves down around you. This is the "curvature."

The paper argues that the AI's learning process (gradient descent) is like a person trying to walk across this trampoline.

The Bias: The AI naturally prefers to walk along the flat, tangent lines because it's easier and requires less energy. It's like rolling a ball down a flat hallway versus trying to roll it up a curved hill.
The Frequency Effect: Because common words are "pinned" so tightly (as mentioned in Analogy 1), they force the AI to focus almost entirely on these flat, easy paths.
The Feedback Loop: As the AI learns, it keeps reinforcing these flat paths. It ignores the curved, complex parts of the trampoline because the "common words" are so dominant.

The Metaphor: Imagine a river. The water (the AI's learning) naturally flows down the path of least resistance (the tangent). Over time, the river carves a deep, narrow channel. The "anisotropy" is just that deep channel. The paper suggests this channel is actually helpful because it simplifies the world, making it easier for the AI to generalize (make good guesses) without getting confused by every tiny detail.

The "Aha!" Moment: Is This Bad?

For years, scientists thought this "squished corner" (anisotropy) was a disease that needed curing. They tried to force the AI to spread out more evenly (isotropy).

This paper says: "Stop! It might be a superpower."

Think of it like this:

Isotropy (Even spread): Imagine a library where every book is scattered randomly on the floor. It's "even," but you can't find anything.
Anisotropy (Squished corner): Imagine the library where all the most popular books are stacked neatly on a single, easy-to-reach shelf. It looks "unbalanced," but it's actually efficient.

The paper suggests that by focusing on these "tangent" directions (the easy, flat paths), the AI is effectively compressing its knowledge. It's ignoring the noise and focusing on the most important patterns. This helps the AI learn faster and generalize better, even though it looks "squished" geometrically.

The Evidence: How They Proved It

The researchers didn't just guess; they watched the AI learn in real-time (like a time-lapse video).

They tracked the "Frequent Flyers": They showed that common words really do get stuck in a tight, narrow space, while rare words wander.
They checked the "Muscle Memory": They looked at the AI's "gradients" (the signals that tell the AI how to change). They found that these signals were almost entirely pushing the AI along the "flat" paths (tangents) and ignoring the "curved" paths.
They tested many models: They checked this on different types of AI (some that read, some that write) and found the same pattern everywhere.

The Takeaway

Don't panic about the "squished" geometry.

The paper concludes that the "anisotropy" in AI models isn't a flaw. It's a natural result of how the AI learns from language. Because we use common words so much, the AI builds a "highway" for them. This highway (the tangent direction) is where the magic happens. It's a form of natural compression that helps the AI make sense of the world by focusing on the most important, frequent patterns rather than getting lost in the details.

In short: The AI isn't broken; it's just taking the most efficient route through the forest, leaving a well-worn path behind it.

1. Problem Statement

Transformer-based language models exhibit a well-documented phenomenon known as anisotropy: token embeddings do not spread uniformly in the high-dimensional vector space but instead concentrate in a narrow cone, resulting in high pairwise cosine similarity even for semantically unrelated tokens.

While previous research has largely treated anisotropy as a "pathology" that degrades semantic reasoning and complicates probing, recent literature suggests it may be an inherent feature of the self-attention mechanism or a form of implicit dimensionality reduction beneficial for generalization. However, existing theoretical explanations often lack a rigorous geometric foundation connecting the training dynamics (specifically frequency bias) to the resulting representation geometry. The paper aims to bridge this gap by deriving geometric arguments for why and how anisotropy emerges during the learning process.

2. Methodology

The authors employ a hybrid approach combining differential geometry theory with mechanistic interpretability experiments conducted during training (rather than just post-hoc).

A. Theoretical Framework

The paper models token embeddings as samples from a local differentiable manifold $\mathcal{M}$ embedded in the ambient Euclidean space $\mathbb{R}^d$ .

Frequency-Induced Variance Collapse: The authors hypothesize that high-frequency tokens have smaller embedding norms and tighter sampling distributions (lower variance) compared to rare tokens.
Curvature Bias Attenuation: Using the chord-arc expansion (relating intrinsic geodesic distance $r$ to ambient Euclidean distance $t$ ), they prove that while curvature naturally biases sampling toward high-curvature directions, a radial concentration of the sampling density (caused by frequent tokens clustering near the origin) attenuates this bias. This effectively suppresses the visibility of high-curvature directions in the marginal distribution.
Gradient Bias: They derive that gradient updates for linear layers are parametrically larger in tangent directions ( $O(t)$ ) compared to normal directions ( $O(t^2)$ ). This creates a self-reinforcing loop where tangent directions are amplified, and normal (curvature) directions are starved of updates.

B. Empirical Validation Protocol

To test these theories, the authors developed a concept-based mechanistic interpretability framework:

Tangent Proxy Extraction: Instead of assuming a global manifold, they extract a low-rank subspace ( $Q_T$ ) from early training activations (pooled across checkpoints). This serves as a proxy for the dominant tangent directions of the local manifold.
Gradient Energy Analysis: They compute the true backpropagated gradients ( $\nabla W$ ) during training and measure the energy concentration within the tangent proxy ( $Q_T$ ) versus matched-rank random normal subspaces ( $Q_N$ ).
Anisotropy Metrics: They use IsoScore* to measure how much anisotropy is removed when projecting out the tangent subspace compared to removing a normal subspace.
Models & Data: Experiments span 10 models (encoders like EuroBERT/CamemBERT and decoders like Pythia/SmolLM2/OLMo) trained on a merged corpus.

3. Key Contributions

Geometric Derivation of Anisotropy: The paper provides the first theoretical proof that frequency-biased sampling (common tokens having smaller norms) combined with manifold curvature leads to a systematic attenuation of curvature visibility, forcing representations into a low-dimensional linear organization.
Tangent Gradient Dominance: It establishes that training dynamics inherently favor tangent directions over normal directions due to the scaling of gradient magnitudes ( $O(t)$ vs $O(t^2)$ ), creating a feedback loop that amplifies anisotropy early in training.
Dynamic Interpretability: Unlike most studies that analyze static, trained models, this work tracks representation geometry during training, demonstrating that anisotropy is imprinted primarily in the early training phase when tangent-biased signals are strongest.
Reframing Anisotropy: The authors argue that anisotropy should not be viewed solely as a defect but as an implicit structural bias that reduces the effective dimensionality of the problem, potentially aiding generalization in overparameterized models.

4. Key Results

Frequency-Variance Correlation: Empirical data confirms a strong negative correlation between token frequency and embedding variance (distance to centroid). Frequent tokens are "pinned" to their idealized forms with low variance, while rare tokens exhibit higher kinematic freedom.
Gradient Energy Concentration: Across all models and layers, true-gradient energy is significantly concentrated in the activation-derived tangent proxies.
- The ratio of tangent energy to normal energy ( $E_{ratio}$ ) is often 10x to 1000x higher than random baselines.
- This effect is strongest in the early phase of training (first 30%) and persists, though slightly weakened, into the late phase.
Anisotropy Attribution: Removing the tangent proxy ( $Q_T$ ) recovers significantly more isotropy (improves IsoScore*) than removing a matched normal subspace. This confirms that the dominant anisotropic structure aligns with the tangent directions of the local manifold.
Architecture Differences: Decoder models (e.g., Pythia) show stronger tangent alignment and frequency-concentration effects than encoder models (e.g., EuroBERT), consistent with the theory that the autoregressive objective induces stronger radial concentration.

5. Significance and Implications

Theoretical Insight: The paper moves beyond descriptive observations of anisotropy to provide a causal geometric mechanism. It explains why language models converge to anisotropic spaces: it is a natural consequence of frequency-biased sampling interacting with the differential geometry of the loss landscape.
Training Dynamics: It highlights that the "shape" of the representation space is largely determined in the early stages of training. This suggests that interventions to control anisotropy (or leverage it) should focus on the initialization and early optimization phases.
Generalization: By framing anisotropy as an implicit dimensionality reduction mechanism, the work suggests that forcing isotropy (e.g., via post-hoc whitening) might actually harm the model's ability to generalize, as it disrupts the natural low-dimensional structure learned during training.
Methodological Shift: The use of activation-derived tangent proxies during training offers a new tool for mechanistic interpretability, allowing researchers to track the evolution of concept subspaces and gradient alignment in real-time.

In conclusion, the paper posits that anisotropy in Transformers is not a bug but a geometric feature arising from the interplay between token frequency, manifold curvature, and gradient dynamics, serving as a natural regularizer that supports generalization in high-dimensional spaces.