💬 NLP

Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations

This paper demonstrates that the intrinsic dimension (ID) of LLM representations serves as a consistent marker of linguistic complexity, showing that different types of syntactic phenomena elicit distinct and predictable ID profiles across model layers.

Original authors: Marco Baroni, Emily Cheng, Iria de-Dios-Flores, Francesca Franzon

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marco Baroni, Emily Cheng, Iria de-Dios-Flores, Francesca Franzon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a super-intelligent robot processes a complex story. You can’t peek inside its "brain" to see neurons firing, but you can watch how much "mental effort" it seems to be exerting by looking at the complexity of the patterns it creates.

This paper is essentially a study of the "mental gymnastics" performed by Large Language Models (LLMs) like Llama or Mistral when they encounter different types of tricky sentences.

Here is the breakdown of how they did it, using some everyday analogies.

1. The Tool: The "Complexity Thermometer" (Intrinsic Dimension)

The researchers used a mathematical concept called Intrinsic Dimension (ID).

Think of it like this: Imagine you are describing a person.

If you only describe their height, you are using 1 dimension (a simple line).
If you describe their height, weight, hair color, eye color, and personality, you are using many dimensions.

The researchers found that when an LLM processes a simple sentence, its "internal description" is very low-dimensional (simple). But when the sentence gets tricky, the LLM has to use a much more complex, high-dimensional "map" to keep track of everything. ID is like a thermometer that measures how much "mental space" the model is using to solve a problem.

2. The Test: Three Levels of "Brain Teasers"

The scientists threw three specific types of linguistic puzzles at the models to see how their "complexity thermometer" reacted:

The "Stacking Blocks" Test (Coordination vs. Subordination):
- Simple: "I ate an apple and I drank water." (Like laying blocks side-by-side on a floor. Easy!)
- Complex: "I know that you think that he said..." (Like building a tower where each block sits inside another. Harder!)
- Result: The models used much more "mental space" (higher ID) for the tower-building sentences.
The "Memory Juggle" Test (Right-Branching vs. Center-Embedding):
- Simple: "The dog that barked ran away." (The information comes in a straight line.)
- Complex: "The dog the cat chased ran away." (You have to hold the "dog" in your head, then "the cat," then "chased," before you can finish the thought.)
- Result: The models showed a spike in complexity early on, likely because they were working hard to "match" the subjects to their actions across a distance.
The "Double Meaning" Test (Ambiguity):
- Simple: "The mother of the baby who was smiling..." (It's clear who is smiling.)
- Complex: "The playmate of the infant who was smiling..." (Is it the playmate or the infant? The model has to keep both possibilities alive.)
- Result: The models stayed in a "high-complexity" state because they had to keep multiple "mental files" open at once to handle the confusion.

3. The Big Discovery: The "Deep Processing" Peak

The most exciting finding is that all these different models—even though they were built differently—showed a similar pattern.

As information moves through the layers of an LLM (from the "eyes" to the "reasoning center"), there is a specific middle section where the Complexity Thermometer hits its highest peak.

The researchers call this the "Abstraction Phase." It’s the moment in the robot's brain where it stops just looking at words as letters and starts understanding them as complex, nested, and meaningful structures. It is the "Aha!" moment of the machine.

Summary in a Nutshell

If you think of an LLM as a chef:

Simple sentences are like making toast (low dimensions, easy to describe).
Complex sentences are like making a five-course French dinner (high dimensions, requires many moving parts).

This paper proves that we can use math to "see" exactly when the chef starts doing the heavy lifting, proving that these AI models aren't just repeating words—they are building complex internal maps to navigate the intricate architecture of human language.

Technical Summary: Tracing the Complexity Profiles of Different Linguistic Phenomena through the Intrinsic Dimension of LLM Representations

1. Problem Statement

While Large Language Models (LLMs) exhibit remarkable linguistic capabilities, the internal mechanisms by which they process complex syntax and semantics remain poorly understood. Specifically, researchers aim to identify whether the "inner layers" of these models correspond to distinct stages of linguistic processing (e.g., surface-level input vs. deep structural abstraction). A key challenge is to find a metric that can differentiate between various types of linguistic complexity—such as formal complexity (structural depth) and functional complexity (processing/memory effort)—and to determine if these complexities manifest at different stages of the model's architecture.

2. Methodology

The authors propose using Intrinsic Dimension (ID) as a probe for linguistic complexity. The core hypothesis is that the more complex a piece of data is for a representational system, the higher its ID will be.

Core Metric (ID): They use the TwoNN estimator to calculate the ID of the last-token embeddings in the residual stream across model layers.
Linguistic Contrasts: To test the granularity of ID, they designed three controlled minimal-pair datasets:
1. Coordination vs. Subordination: Contrasting flat structures (using and) with hierarchical, recursive structures (using that).
2. Right-branching vs. Center-embedding: Contrasting linear extensions with long-distance dependencies that increase memory load.
3. Unambiguous vs. Ambiguous Attachment: Contrasting sentences where semantic cues resolve relative clause attachment versus those where ambiguity remains.
Models: The study is cross-model, testing six LLMs of varying sizes (7B to 14B parameters), including Llama-3, Gemma-2, Mistral, and Qwen-2.5.
Validation Techniques:
- Representational Similarity: Using Information Imbalance ( $\Delta$ ) to measure the directional divergence between representation spaces.
- Layer Pruning: Iteratively removing layers to observe the impact on next-token prediction accuracy and KL divergence, testing the causal importance of specific layers.

3. Key Contributions

Granular Complexity Mapping: Unlike previous studies that looked at generic corpus data, this work demonstrates that ID can distinguish between specific, well-known (psycho)linguistic complexity contrasts.
Identification of the "Abstraction Phase": The paper confirms that the "generic ID peak" (the point where ID is highest for random text) coincides with the phase where models perform the most complex linguistic tasks, marking a transition to deep structural processing.
Differentiating Formal vs. Functional Complexity: The study provides empirical evidence that different types of complexity trigger different ID profiles and occur at different depths in the model.

4. Results

The results show a remarkable degree of consistency across disparate LLMs:

Subordination vs. Coordination: Subordinated (nested) sentences elicit significantly higher ID profiles. This divergence typically emerges around the "generic ID peak," suggesting that the model's deep layers are specifically dedicated to building hierarchical structures.
Center-embedding vs. Right-branching: The distinction is visible earlier in the layers (pre-peak), likely reflecting lower-level processing of long-distance agreement. Interestingly, right-branching sentences sometimes show higher ID in very late layers, which the authors attribute to the presence of a nested clause at the end of the sentence.
Ambiguity Resolution: Ambiguous sentences maintain a consistently higher ID than unambiguous ones across most layers, suggesting that ambiguity creates a state of "representational instability" rather than a specific phase-based processing step.
Causal Validation: Pruning experiments confirmed that removing layers within the "generic ID peak" span significantly degrades performance on subordinated sentences more than on coordinated ones, proving these layers are crucial for processing nested structures.

5. Significance

This research has dual significance:

For AI Interpretability: It suggests that the "universality" observed in LLMs is not accidental but stems from the fact that different models converge on similar geometric strategies to organize linguistic processing. It provides a tool (ID) to map the "functional anatomy" of a transformer.
For Linguistics: It positions LLMs as useful in silico models for studying human language faculty. The ability of ID to track complexity suggests it could eventually serve as a formal metric to quantify linguistic complexity in ways that complement traditional psycholinguistic measures.