💬 NLP

On the Emergence and Test-Time Use of Structural Information in Large Language Models

This paper investigates how large language models learn and apply abstract structural information from observational data, revealing that while the emergence of such learning correlates with complex reasoning, the models' ability to perform test-time compositional generation remains limited.

Original authors: Michelle Chao Chen, Moritz Miller, Bernhard Schölkopf, Siyuan Guo

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Michelle Chao Chen, Moritz Miller, Bernhard Schölkopf, Siyuan Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Large Language Models (LLMs) as incredibly talented chefs who have read every cookbook in the world. They can recite recipes perfectly and even mix ingredients they've seen before to make new dishes. But can they truly understand the logic of cooking? Can they take a basic rule (like "if you add salt, it tastes salty") and apply it to a completely new ingredient they've never seen, or do they just memorize specific dishes?

This paper, "On the Emergence and Test-Time Use of Structural Information in Large Language Models," is like a scientific kitchen experiment designed to answer that question. The researchers wanted to see if these AI chefs learn the underlying "rules of grammar" (the structure) or just memorize the specific sentences they were trained on.

Here is a breakdown of their findings using simple analogies:

1. The Playground: A "Grammar Gym"

To test the models, the researchers didn't just throw random sentences at them. They built a special "Grammar Gym" based on a concept called Transformational Grammar.

Think of language like a set of Lego blocks.

Deep Structure: The core idea of a sentence (e.g., "John is happy").
Surface Structure: How we actually say it (e.g., "It seems that John is happy").

The researchers created a dataset where the AI had to perform specific "transformations" on sentences, like turning a statement into a question, changing active voice to passive voice, or moving a subject to the front of the sentence. It's like asking the chef to take a cake recipe and instantly rewrite it to be a pie recipe, following strict rules.

2. The Discovery: When Does the AI "Get It"?

The researchers watched the AI as it trained, looking at its internal "brain waves" (mathematical representations) to see when it started understanding these rules.

The "Aha!" Moment: They found that the AI didn't learn these rules gradually. Instead, it went through a phase transition. Imagine a light switch flipping on. At a certain point in training (around step 64,000), the AI suddenly started distinguishing between different types of sentence changes.
The Connection to Reasoning: Interestingly, this "switch" didn't just help the AI predict the next word better. It correlated with the AI getting better at complex reasoning tasks. It suggests that to solve hard logic puzzles, the AI needs to understand the underlying structure of language, not just the words.

3. The Test: Can the AI Combine Rules? (The "Composition" Problem)

This is the most critical part of the experiment. The researchers asked: If we teach the AI Rule A and Rule B separately, can it combine them to do Rule A + Rule B on a sentence it has never seen before?

The Result: The AI struggled.
- When the AI was given the intermediate step (e.g., "First do Rule A, then do Rule B"), it did well.
- But when asked to do the whole chain in one go without help, it often failed.
The Metaphor: Imagine teaching a student how to tie their left shoe and how to tie their right shoe. If you ask them to tie both at the same time without looking at the steps, they might get confused. The AI seems to memorize the specific combination of steps rather than truly understanding the independent rules well enough to mix and match them freely.

4. Inside the Machine: Where Does the Magic Happen?

The researchers also performed "surgery" on the AI's brain (called ablation studies) to see which parts were doing the heavy lifting.

The "Muscle" vs. The "Brain": They found that the MLP layers (think of these as the AI's "muscles" or processing units that do the heavy calculation) were responsible for about 65% of the success in these transformations. The Attention heads (the "brain" that decides what to focus on) only did about 35%.
The Final Layer: The actual decision of "how to transform this sentence" happens mostly in the final layers of the network. It's like the AI spends most of its time thinking, and then in the very last second, it figures out the specific rule to apply.

5. The Bottom Line

The paper concludes with a mix of good news and a reality check:

Good News: LLMs do learn structural information. They don't just memorize text; they develop an internal map of how sentences work, and this map gets sharper as they get better at reasoning.
Reality Check: However, this learning isn't perfect yet. When it comes to combining these rules to generate genuinely new, complex knowledge on the fly (test-time composition), the AI is still limited. It relies heavily on seeing intermediate steps and struggles to "disentangle" the rules to apply them flexibly in new situations.

In short: The AI is learning the rules of the game, but it's still a bit like a student who knows the math formulas but needs to see the example problem solved step-by-step before they can solve a new one on their own. They haven't quite mastered the art of pure, flexible creativity yet.

Technical Summary: On the Emergence and Test-Time Use of Structural Information in Large Language Models

Problem Motivation
The paper addresses the capacity of Large Language Models (LLMs) to learn abstract structural information from observational data and utilize this knowledge to generate genuinely new combinations at test-time. While humans can reorganize basic linguistic building blocks to comprehend unseen combinations (compositional generalization), current models often struggle with systematic generalization to out-of-distribution (OOD) concepts. The authors posit that understanding how LLMs learn and compose structural information is central to producing new knowledge outside the training corpus, a capability linked to mechanistic understanding and robust prediction. To study this in a controlled setting, the authors move beyond standard benchmarks to focus on linguistic structural transformations derived from Transformational Grammar.

Methodology
The authors introduce a controlled synthetic playground based on Transformational Grammar (TG).

Dataset Construction: They generated a natural language dataset comprising ten distinct transformation types (e.g., NP-raising, passivization, I-movement) grouped into five syntactic categories. The dataset includes:
- Single-level transformations: Applying one operator to a base sentence.
- Nested transformations: Sequential application of operators (e.g., $B(A(s))$ ).
- Compositional ambiguity: Cases where certain transformations cannot be meaningfully composed, requiring the model to implicitly identify compatible patterns.
- The dataset was generated using DeepSeek-V3 with specific prompts to ensure consistency, yielding ~2000 samples for single-level and ~500 for nested transformations.
Experimental Setup:
- Emergence Analysis: Using the Pythia-410M model, the authors tracked the evolution of structural representations during training. They computed the $L_2$ distance between the residual stream representations of base sentences and their transformed versions ( $\Delta A(s)$ ) to measure structural sensitivity.
- Compositional Generalization: They evaluated LLaMA3-8B using both full-parameter fine-tuning and parameter-efficient fine-tuning (LoRA). The evaluation tested the model's ability to perform nested transformations and OOD combinations (e.g., training on $A(B(s))$ and $B(C(s))$ , then testing on $A(C(s))$ ) without intermediate steps provided during inference.
- Mechanistic Interpretability: The authors performed causal intervention analysis (zeroing attention heads and MLP blocks) and linear probing (LDA) to identify which network components and layers are responsible for syntactic transformations.

Key Contributions

Dataset: Introduction of a natural language dataset based on linguistic structural transformations to formally study structural information learning.
Empirical Correlations: Demonstration that the emergence of structural information during learning correlates with success in complex reasoning tasks, rather than just standard language modeling perplexity.
Limitations in Composition: Evidence that while models learn structural representations, their ability to perform test-time compositional generation (specifically for unseen nested transformations) remains limited under fine-tuning.
Mechanistic Insights: Identification of specific network components responsible for structural learning, revealing a concentration of transformation-relevant computation in late layers and a dominance of MLP blocks over attention mechanisms.

Results

Emergence of Structure:
- Different sentence transformations form distinct, separable clusters in the embedding space.
- The $L_2$ norm between base and transformed sentence representations increases steadily during training, with a sharp rise around step 64k, indicating a phase transition where structural distinctions become pronounced.
- This structural refinement correlates more closely with gains in complex reasoning tasks than with standard next-word prediction perplexity.
Compositional Generalization:
- Full-parameter fine-tuning performed poorly, likely due to overfitting on the limited dataset.
- LoRA fine-tuning showed significant improvement over full-parameter tuning. However, the model struggled with exact matches for nested transformations when intermediate results were not provided during inference.
- Performance on OOD nested transformations (e.g., $A+C$ ) remained low (0.00% exact match, ~45-49% partial match), suggesting the model does not fully disentangle transformation operations to compose them flexibly at test-time.
- Providing explicit intermediate steps during training and inference improved performance, indicating the model benefits from structural guidance but cannot yet robustly generalize to unseen compositions without it.
Ablation and Interpretability:
- Layer Concentration: The probability of predicting the correct transformation token increases sharply in the final third of the network (layers 24–32), with 38% of the contribution concentrated there.
- Component Preference: MLP blocks contribute more to syntactic transformation success (65% of causal contribution) than attention mechanisms (35%).
- Linear Separability: Linear probes show near-perfect separation between transformation types in the residual stream, supporting the linear representation hypothesis.

Significance and Claims
The paper claims to provide evidence regarding the limits and mechanisms of LLMs in generating new knowledge through structural composition. The authors conclude that while LLMs can learn structural information and exhibit a phase transition in structural sensitivity during training, they currently lack the ability to flexibly re-compose out-of-distribution structures to produce new knowledge without explicit intermediate guidance. The findings suggest that reasoning may require structural information, but current fine-tuning regimes on models like LLaMA3-8B do not yet achieve robust compositional generalization. The work highlights that syntactic transformations rely heavily on late-layer MLPs and that the separation of concepts emerges linearly in the residual stream as fine-tuning progresses.

Limitations
The authors acknowledge that their analysis is limited to English, potentially restricting generalizability to cross-linguistic patterns. The use of single-level transformations for causal attribution may conflate sub-operations. Furthermore, findings on an 8B model may not scale directly to larger models, which would require more complex datasets to avoid memorization.