💬 NLP

Probing Minimalist Phase Structure in LLMs: What Universal Dependencies Cannot Represent

This paper demonstrates that large language models encode formal-syntactic abstractions like Minimalist Program phase boundaries and cohesion—concepts invisible to Universal Dependencies—by revealing that structural probes detect phase-count gradients and sign asymmetries in wh-movement stimuli that standard UD-based probing cannot capture.

Original authors: Yuanhao Chen, Peter Chin

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuanhao Chen, Peter Chin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out how a giant, super-smart robot (a Large Language Model or LLM) understands the structure of a sentence. For years, researchers have used a "map" called Universal Dependencies (UD) to check this. Think of UD as a standard, flat map that shows how words connect to each other, like a family tree. It tells you who is the "parent" and who is the "child" in a sentence.

However, the authors of this paper argue that this map is missing some crucial terrain. It doesn't show the "walls" or "boundaries" between different rooms in a house. In linguistics, these walls are called phases.

Here is the core story of the paper, broken down simply:

1. The Problem: The Map is Too Simple

The researchers wanted to know if these AI robots actually understand these invisible "walls" (phases) or if they only see the flat map (UD).

The Trap: If you just ask the robot to draw a sentence, it might look like it understands the walls. But maybe it's just memorizing the flat map.
The Solution: The team built a special test where the "flat map" (UD) looks exactly the same for three different types of sentences. If the robot still reacts differently to these sentences, it means it sees something the map doesn't show.

2. The Experiment: Three Types of "Rooms"

They created three types of sentences involving a question word (like "What") moving from the end of the sentence to the front.

Type A (Bare): A small, open room. (e.g., "What did she see him eat?")
Type B (Infinitival): A room with a hallway. (e.g., "What did she expect him to eat?")
Type C (Finite): A room with a hallway and a locked door. (e.g., "What did she think he ate?")

In all three cases, the "flat map" distance between the question word and the subject is identical. But in the "Minimalist" theory of grammar, the number of "walls" (phases) the question word has to cross increases from Type A to Type C.

3. The Findings: The Robot Sees the Walls

The researchers "probed" 13 different AI models (from families like Llama, Gemma, Mistral, and Qwen) to see how they represented these sentences.

The "Depth" Test: They measured the distance between the question word and the subject.
- Result: The robots consistently treated the sentences as if they were getting "deeper" or more complex as they added more "walls." Even though the flat map said the distance was the same, the robot's internal math showed a gradient: Simple < Medium < Complex.
- The Stat: 12 out of 13 models showed this pattern clearly.
The "Cohesion" Test (The Big Surprise): They looked at the relationship between the subject and the verb inside the small room.
- The Prediction: In the "Finite" sentence (Type C), the subject and verb are in the same "room" (phase). In the others, they are separated by a wall.
- The Result: The robots pulled the subject and verb closer together in their internal memory only for the Finite sentence.
- Why this matters: The flat map says the distance is the same (1 step) for all three. The robots shouldn't have noticed a difference. But they did. They treated the words in the "Finite" room as a tight-knit team, while the others were looser. This proves they are using a concept called phase-internal cohesion, which the standard map cannot see.

4. Proving It's Real (Not Just a Glitch)

Skeptics might say, "Maybe the robot is just guessing, or the numbers are random." To prove the robot is actually using this information, the researchers did a "surgery" (called activation patching).

They took the "brain state" of the subject word from a complex sentence and pasted it into a simple sentence.
Result: The robot's internal distance measurements changed exactly as predicted. This confirms the robot isn't just memorizing patterns; it is actively using these structural "walls" to do its math.

5. The Main Takeaway

The paper concludes that Universal Dependencies (the standard map) is a "lower bound," not an "upper bound."

Think of it like this: If you check a robot's knowledge using a flat map, you might think it knows the basics. But this paper shows the robot actually knows the architecture of the building—the walls, the floors, and the rooms—even though the map doesn't draw them.

In short: Large Language Models, through their training, have spontaneously learned to build internal maps that align with deep, formal linguistic theories about "phases" and "boundaries," going far beyond what standard annotation tools can detect. They aren't just mimicking surface patterns; they are building a structural understanding of language that includes invisible walls.

Technical Summary: Probing Minimalist Phase Structure in LLMs

Problem Statement
Structural probing has established that Large Language Models (LLMs) encode syntactic structure in their hidden representations, typically trained to recover Universal Dependencies (UD) tree distances. However, UD is an annotation scheme, not a generative grammar; it lacks formal-syntactic abstractions such as phase boundaries (from the Minimalist Program) or phase-internal cohesion. Consequently, standard UD-based probing cannot determine whether LLMs encode these deeper structural properties. If a probe detects a difference between conditions, it is unclear whether this reflects the target syntactic abstraction or merely a re-encoding of UD distances or surface properties. This paper addresses the open question of whether LLMs encode formal-syntactic abstractions beyond the reach of annotation-based probing.

Methodology
The authors design a probing experiment using wh-movement stimuli where the UD tree distance between probed word pairs is held invariant across conditions by construction. This ensures that any non-zero effect detected by the probe reflects structural information not captured by UD.

Stimuli Construction:
Three conditions are created for each item, ordered by the number of Minimalist Program (MP) phase boundaries the wh-element crosses:
- Bare Small Clause: (e.g., What did she see him eat?) – 0 additional phases.
- Infinitival: (e.g., What did she expect him to eat?) – 1 additional phase (vP).
- Finite: (e.g., What did she think he ate?) – 2 additional phases (vP and CP).
Two probe pairs are measured:
- Cross-clause (wh–embedded subject): Tests the "phase-count gradient" (structural depth).
- Within-clause (embedded subject–embedded verb): Tests "phase-internal cohesion." Crucially, the UD distance for this pair is exactly 1 edge (nsubj) in all three conditions.
Probing Protocol:
- Linear probes are trained on the Universal Dependencies English Web Treebank (UD-EWT) to map hidden states to Euclidean space.
- These trained probes are evaluated on the wh-movement stimuli without further fitting.
- Effect sizes ( $\beta$ ) are estimated via OLS regression comparing Finite and Infinitival conditions against the Bare baseline.
- Canonical-Layer Reporting: To avoid the opportunistic selection of layers (a common issue where different contrasts peak at different layers), the authors define a single "canonical layer" ( $L^*$ ) per model based on the layer maximizing the most reliable contrast ( $\beta_{fin}$ ). All other contrasts are reported at this fixed layer.
Causal Verification:
An activation patching experiment is conducted. The residual-stream representation of the embedded subject from an Infinitival source is patched into a Bare target. If the representation causally encodes clause-type information, this should shift the probe distance in the predicted direction.
Models:
The study evaluates 13 decoder-only base models from four families: Gemma-3, Llama-3, Mistral, and Qwen.

Key Results

Phase-Count Gradient (Cross-Clause):
Across 12 of 13 models, the probe distance between the wh-element and the embedded subject follows the predicted gradient: $\beta_{fin} > \beta_{inf} > 0$ . This indicates that the model representations reflect the increasing number of phase boundaries crossed, even though UD distances are invariant.
Sign Asymmetry (Within-Clause):
On the embedded subject–verb pair (where UD distance is identical), a robust 13/13 sign asymmetry is observed:
- Finite condition: $\beta < 0$ (representations are closer than baseline).
- Infinitival condition: $\beta > 0$ (representations are further than baseline).
This pattern contradicts alternative accounts:
- UD-distance: Predicts $\beta \approx 0$ (distance is constant).
- Linear distance: Predicts $\beta_{inf} > 0$ but $\beta_{fin} \approx 0$ (finite and bare are adjacent; infinitival has an intervening 'to').
- Monotone complexity: Predicts $\beta_{fin} > \beta_{inf} > 0$ (more structure = larger distance).
The observed negative $\beta_{fin}$ is specifically predicted by phase-internal cohesion, an MP abstraction where items within the same completed phase (Finite CP) are spelled out as a cohesive unit, making them representationally closer.
Causal Corroboration:
Activation patching at the embedded subject position successfully shifts probe distances in the predicted direction for 12 of 13 models. This confirms that the representations identified are causally active in the model's computation, addressing skepticism regarding whether probes recover representations the model actually uses.

Key Contributions

Invariant UD Distance Probing: The authors demonstrate a method to probe for syntactic structure where UD distances are strictly controlled, ensuring that detected effects reflect structure beyond UD.
Canonical-Layer Reporting: They introduce a stricter reporting standard that anchors all contrasts to a single layer, removing a degree of freedom and revealing heterogeneity (e.g., the Qwen-3-4B failure) that per-contrast peak reporting would conceal.
Causal Evidence: Through activation patching, they provide direct evidence that the phase-structure representations are computationally active, not merely correlational.

Significance and Claims
The paper claims that distributional pretraining induces representations aligned with formal-syntactic abstractions (specifically phase boundaries and phase-internal cohesion) that are invisible to UD-based annotation. The 13/13 sign asymmetry on the within-clause pair serves as an existence proof that LLMs encode structural information inaccessible to standard probing.

The authors conclude that UD-grounded probes provide a lower bound on syntactic encoding, not an upper bound. The findings suggest that LLMs may possess a richer syntactic representation than what is captured by standard dependency annotations, driven by the statistical regularities of the training data rather than explicit grammatical instruction.

Limitations
The study is limited to English and specific verb classes. The authors note that the phase-count gradient is directional (predicting order, not magnitude ratios) and that scaling effects within model families are heterogeneous. The causal claim is conservative, limited to the embedded subject's representation affecting probe distances generally, rather than proving specific mechanisms for the sign asymmetry.