💬 NLP

Local and Global Regimes of Geometric Complexity in Language Model Representations

This paper reveals that the relationship between lexical diversity and intrinsic dimensionality in language model representations undergoes a predictable, scale-dependent reversal, demonstrating that intrinsic dimensionality is not a straightforward measure of complexity but rather reflects a fundamental organizational principle of linguistic data.

Original authors: Arwa Osman, Marco Baroni, Iuri Macocco

Published 2026-08-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Arwa Osman, Marco Baroni, Iuri Macocco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand the shape of a massive, invisible cloud made of pure thought. This is the world of Artificial Intelligence, specifically the "brain" of a Large Language Model (LLM). These models don't just store words; they turn every sentence they read into a complex, multi-dimensional map. To figure out how complicated these maps are, scientists use a tool called Intrinsic Dimensionality (ID). Think of ID as a way to measure how "spread out" or "crowded" the data is. If a group of ideas is very simple, they might all huddle together in a tight, flat circle (low ID). If they are wildly complex and diverse, they might stretch out into a sprawling, high-dimensional jungle (high ID). Scientists love using ID to guess how smart a model is or how hard a task is, assuming that a higher number means a richer, more complex structure. But here's the catch: what if the number isn't telling us about the ideas at all, but just about how we counted them?

This is the puzzle tackled by Arwa Osman, Marco Baroni, and Iuri Macocco in their paper, "Local and Global Regimes of Geometric Complexity in Language Model Representations." They discovered that the "complexity score" (ID) of a language model's brain isn't a fixed truth. Instead, it flips like a switch depending on how many different words you feed it versus how many times you repeat them. They found that if you have a small crowd of unique words, the model looks surprisingly complex. But if you have a huge crowd of unique words, the model looks even more complex—but for a completely different reason. The most surprising part? If you compare two types of words (like "nouns" and "prepositions") without controlling for how many unique words are in each group, you might get the answer backward. It turns out that the "complexity" we see might just be an optical illusion caused by the size of the vocabulary, not the actual difficulty of the words themselves.

The Great Vocabulary Switcheroo

To understand this, let's imagine the language model's brain as a giant, magical dance floor. Every time the model sees a word, it sends a dancer onto the floor to strike a pose. If the model sees the word "cat" a thousand times, it sends out a thousand "cat" dancers. Even though they are all "cat" dancers, they might pose slightly differently depending on the sentence they are in (e.g., "The cat sleeps" vs. "The cat jumps").

The researchers wanted to know: How does the number of different words (lexical diversity) change the shape of this dance floor?

They set up a massive experiment using a dataset of 10,000 samples. They created 11 different scenarios. In one scenario, they used just 1 unique noun (like "cat") repeated 10,000 times. In another, they used 10,000 unique nouns (like "cat," "dog," "elephant," etc.), with each word appearing only once. They then measured the "Intrinsic Dimensionality" (ID) of the dancers' poses using a special ruler called GRIDE.

The Two Regimes: The Local vs. The Global

The results were wild. The relationship between the number of unique words and the complexity score didn't just go up or down; it reversed depending on the scale of the measurement.

1. The Local Regime (The "Crowded Room" Effect)
When the researchers used a small measurement scale (looking at just a few neighbors at a time) and had low lexical diversity (few unique words, many repeats), something strange happened. The fewer unique words they had, the higher the complexity score.

The Analogy: Imagine a dance floor with only one type of dancer (say, "cats"). If you have 10,000 cats, they are all crowded together. But because there are so many of them, they are forced to find every tiny, subtle way to pose differently to avoid bumping into each other. The "local" area around any single cat is packed with variations. The ruler sees this dense, crowded variation and says, "Wow, this is a very complex, high-dimensional space!"
The Result: Fewer unique words = Higher ID (in this specific local view).

2. The Global Regime (The "Massive Parade" Effect)
When they switched to a larger measurement scale or increased the number of unique words, the rule flipped. Now, having more unique words led to a higher complexity score.

The Analogy: Now imagine a parade with 10,000 different types of dancers (cats, dogs, elephants, astronauts, etc.). Each type has its own distinct spot on the dance floor. The "local" area around any single dancer is empty because there are no other "cats" nearby to crowd them. Instead, the ruler looks at the whole parade and sees that the dancers are spread out across the entire universe of possibilities. The space is huge because there are so many different types of things.
The Result: More unique words = Higher ID (in this global view).

The Magic Switch Point

The most exciting part of the paper is that the authors didn't just observe this flip; they predicted exactly where it would happen.

They derived a simple, perfect formula for the "tipping point" where the rule switches from "fewer words = higher ID" to "more words = higher ID." The switch happens when the number of unique words ( $n$ ) equals the total number of samples ( $N$ ) divided by the measurement scale ( $k$ ).

The Formula: $n = N / k$
What it means: If you are looking at a neighborhood of 128 dancers ( $k=128$ ) and you have 10,000 total samples ( $N=10,000$ ), the switch happens when you have exactly 78 unique words ( $10,000 / 128 \approx 78$ ).
The Proof: They tested this on two different giant AI models (Qwen3-8B and Meta-Llama-3-8B) and found that the switch happened exactly at this predicted number every single time. It wasn't a guess; it was a mathematical certainty based on how the data is arranged.

Why This Matters (And Why We Were Wrong)

This discovery is a huge warning label for scientists who use these tools. The paper shows that if you compare two groups of words without controlling for how many unique words are in each group, you might get the answer completely backward.

The "Noun vs. Preposition" Trap:
The authors showed a classic example. In a natural text, "nouns" (like dog, house, idea) are thousands of unique words, while "prepositions" (like in, on, at) are only a few dozen.

The Standard View: If you just measure the complexity of nouns vs. prepositions, nouns look like they live in a much more complex, high-dimensional space. You might think, "Nouns are so rich and varied!"
The Reality: The authors matched the two groups so they both had exactly 25 unique words. Suddenly, the result flipped! The prepositions now looked more complex than the nouns.
The Lesson: The original difference wasn't because nouns are "better" or "more complex." It was an artefact (a glitch) caused by the fact that nouns had thousands of unique types and prepositions only had a few. The measurement tool was just reacting to the count of unique words, not the nature of the words.

The Takeaway

This paper teaches us that when we look at the "shape" of an AI's brain, we have to be very careful about what we are actually measuring.

If you are looking at a small group of repeated words, the complexity score tells you how much the model varies its pose for that one word in different contexts.
If you are looking at a huge group of unique words, the complexity score tells you how spread out the entire vocabulary is.

These are two different things. You can't compare them directly without realizing you are measuring two different regimes. The authors found a precise mathematical rule to tell you which regime you are in. It's a reminder that in the world of AI, sometimes the most "complex" looking numbers are just a reflection of how we set up the experiment, not a deep secret of the machine's mind. The "truth" of the model's geometry depends entirely on the scale at which you choose to look.

Technical Summary: Local and Global Regimes of Geometric Complexity in Language Model Representations

Problem Statement

Intrinsic dimensionality (ID) has become a standard metric for probing the representational complexity of language models (LMs), often used to infer properties of linguistic structures (e.g., whether nouns occupy higher-dimensional spaces than prepositions). However, it remains unclear whether observed ID differences reflect genuine linguistic properties or are merely artifacts of dataset construction, specifically lexical diversity (the number of unique word types).

The paper highlights a critical confound: comparisons between word classes with vastly different natural vocabulary sizes (e.g., open-class nouns vs. closed-class prepositions) without controlling for lexical diversity yield misleading ID results. When lexical diversity is controlled, these results can reverse, suggesting that standard ID estimators may be measuring dataset properties rather than the intrinsic geometric complexity of the representations themselves.

Methodology

The authors isolate lexical diversity as a controlled experimental variable to investigate its impact on ID estimates.

Dataset Construction: Using WikiText-103, the authors constructed 11 distinct conditions, each containing exactly $N = 10,000$ samples. These conditions varied systematically in the number of unique target nouns ( $n$ ), ranging from 1 unique noun (repeated 10,000 times) to 10,000 unique nouns (appearing once each). Crucially, the context preceding the target word was drawn independently from the corpus, ensuring that lexical diversity was isolated to the target token's identity.
Models: Experiments were conducted on Qwen3-8B (primary) and Meta-Llama-3-8B (for cross-architecture validation).
ID Estimation: The study employed the Generalized Ratios Intrinsic Dimension Estimator (GRIDE) across six neighborhood scales ( $k \in \{16, 32, 64, 128, 256, 512\}$ ). GRIDE estimates ID based on the distribution of distance ratios between nearest neighbors.
Validation: The analysis was replicated using the Maximum Likelihood Estimator (MLE) and a Part-of-Speech (POS) diversified dataset (including verbs, adjectives, etc.) to ensure the findings were not specific to nouns or the GRIDE algorithm.

Key Findings and Results

1. A Scale-Dependent Transition

The study reveals a sharp transition between two distinct regimes of ID behavior, determined by the relationship between the neighborhood scale ( $k$ ) and the number of samples per word ( $m = N/n$ ).

Local Regime ( $k < m$ ): When the neighborhood scale is smaller than the number of samples per word, the estimator samples entirely within the "cloud" of a single word type. In this regime, ID is inversely related to lexical diversity: conditions with fewer unique words (and thus more samples per word) exhibit higher ID. This reflects the model's ability to capture the contextual variability of individual lexical items.
Global Regime ( $k \ge m$ ): When the scale exceeds the number of samples per word, the neighborhood inevitably includes different word types. In this regime, ID is directly related to lexical diversity: conditions with more unique words exhibit higher ID. This reflects the geometric spread of the entire vocabulary manifold.

2. The Exact Transition Point

The authors derive a parameter-free formula for the transition point where the ID ordering reverses:
$n_{transition} = \frac{N}{k}$
Empirical results confirm that this transition occurs exactly at this predicted point across all tested scales, models, and layers. When ID curves are plotted against the ratio $k/m$ , data from different scales collapse onto a single trajectory, crossing the transition at $k/m = 1$ .

3. Geometric Interpretation

Local Regime: The estimator measures the "richness" of the local geometry around a specific word type. More samples allow for a better estimation of the word's contextual variance, leading to higher ID estimates for low-diversity conditions.
Global Regime: The estimator measures the geometry of the broader noun space. As the number of unique words increases, the estimator samples more independent directions in the representation space, leading to higher ID estimates for high-diversity conditions.
Saturation: In the global regime, ID eventually saturates as the "global ratio" (the fraction of neighbors belonging to different word types) approaches 1. Beyond a certain lexical diversity, adding more unique words yields diminishing returns in ID estimation.

4. Reversal of Apparent Linguistic Effects

The paper demonstrates that the "Noun vs. Preposition" ID difference observed in comparisons without controlling for lexical diversity (where nouns show higher ID) is an artifact of lexical diversity. When the two classes are matched for vocabulary size (e.g., 25 unique types each), the relationship reverses, with prepositions showing higher ID. This confirms that the original difference was driven by dataset construction, not syntactic category.

Significance and Claims

The paper claims that ID estimates are not absolute measures of complexity but are highly sensitive to the interplay between dataset construction (lexical diversity) and the scale of the estimator.

Caution in Interpretation: Researchers must exercise extreme care when interpreting ID differences across categories (e.g., word classes, languages, or domains). If categories differ in lexical diversity, ID comparisons may be invalid unless the conditions fall within the same regime (local or global).
Diagnostic Tools: The authors provide a formula ( $n_{transition} = N/k$ ) and a "same-word fraction" diagnostic to determine which regime a specific experimental condition occupies. This allows researchers to verify whether they are measuring local contextual variability or global manifold structure.
Fractal-like Structure: The observation that ID profiles (such as the mid-layer "peak" or "hump") appear in both regimes suggests a "fractal-like" organization of linguistic data in LLMs, where similar geometric patterns emerge at both narrow (local) and wide (global) scales.
General Principle: The discovery of these two regimes reveals a general principle of how linguistic data is organized in LLMs, shedding new light on the inner manifold structures of these models.

The authors conclude that while ID is a valuable probe, its values depend on the "view" of the manifold provided by the specific scale and dataset composition, and comparisons across different regimes are not straightforward.