🤖 AI

Emergent Semantic Role Understanding in Language Models

This paper demonstrates that semantic role understanding partially emerges in frozen decoder-only transformers during pre-training, as evidenced by linear probes, though full performance requires fine-tuning and the internal representation of these roles becomes increasingly distributed as model scale increases.

Original authors: Carla Griffiths, Mirco Musolesi

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Carla Griffiths, Mirco Musolesi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a child how to understand stories. You have two choices:

The "Read and Learn" Method: Let the child read thousands of books on their own, hoping they naturally figure out who did what to whom.
The "Drill" Method: After they read, you sit them down and explicitly teach them, "In this sentence, the dog is the doer, and the ball is the thing being kicked."

This paper asks a big question: Do large language models (the "children") figure out the "who did what to whom" logic just by reading (pre-training), or do they need the explicit drills (fine-tuning) to learn it?

The researchers, Carla Griffiths and Mirco Musolesi, set up an experiment to find out. Here is how they did it and what they discovered, explained simply.

The Experiment: The "Frozen Brain" Test

To see if the model learned this logic on its own, the researchers used a clever trick. They took a model that had finished reading its training data (pre-training) and froze its brain. They didn't allow the brain to change or learn anything new.

Then, they attached a very simple, tiny "probe" (like a basic question-answering tool) to the frozen brain and asked it to identify roles in sentences (e.g., "Who took the walk?").

If the frozen brain could answer well: It means the model learned the logic just by reading.
If the frozen brain failed: It means the model only learned the logic when it was later "drilled" with specific tasks (fine-tuning).

They tested this on models of four different sizes, from a tiny "puppy" (0.4 million parameters) to a "medium-sized dog" (57 million parameters).

The Findings: What Did They Discover?

1. The Logic is Already There (But Not Perfect)

The researchers found that even the frozen brains could answer the questions much better than a random guess.

The Analogy: Imagine a student who has read a library of books but never took a test. When you hand them a simple quiz, they get about 60% of the answers right just from their reading. They didn't need the teacher to explain the rules; the rules were already in their head.
The Result: Semantic role understanding (knowing "who did what") does emerge during the pre-training phase. It is not something that only appears after specific training.

2. Bigger Brains Need More "Drills"

Here is the twist: As the models got bigger, the gap between the "frozen brain" and the "fully trained brain" actually got slightly wider.

The Analogy: Think of the tiny model as a student who memorized the rules perfectly just by reading. The big model is like a genius student who read the library, but their brain is so complex and full of other information that they need a little bit of specific coaching to organize that knowledge efficiently for a test.
The Result: While the big models had the information, they needed fine-tuning to unlock it fully. The "drills" helped them organize their massive knowledge base better.

3. The "Who Did What" Neurons

The researchers looked inside the models to see how they stored this information. They found specific groups of neurons (tiny processing units) that acted like specialists.

The "Time" Specialists: Some neurons were like dedicated timekeepers. No matter how big the model got, these neurons were always crucial for understanding when things happened.
The "Doer" (Agent) Specialists: This was the most surprising part.
- In small models, specific neurons were the "Doer" experts. If you turned them off, the model forgot who did what.
- In large models, these same "Doer" neurons actually started to get in the way! If you turned them off in a big model, the model actually got better at its job.
The Analogy: In a small team, you need one specific person to be the "Team Leader." If you fire them, the team collapses. But in a massive corporation, having one person try to be the sole leader might cause bottlenecks. The big company works better if the leadership is spread out among many people. The big models shifted from having a single "Doer" neuron to a distributed network where the job is shared.

The Bottom Line

This paper proves that language models learn the basic structure of "who did what to whom" just by reading text, without needing specific instruction. This knowledge is "baked in" during pre-training.

However, as models get bigger, they don't just get "smarter" in a straight line. They change how they store this knowledge. They move from relying on a few specific "specialist" neurons to using a complex, distributed network. This means that what works for a small model (like relying on one specific neuron) doesn't necessarily work for a giant model, which has reorganized its internal logic to handle its massive size.

In short: The models learned the rules of the game just by watching it played (pre-training), but as they grew up, they had to change their playing style to win.

Technical Summary: Emergent Semantic Role Understanding in Language Models

Problem Statement

The central problem addressed is the ambiguity surrounding the origin of linguistic capabilities in Large Language Models (LLMs). Specifically, it remains unclear whether semantic role understanding (SRL)—the ability to identify "who did what to whom" (predicate-argument structure)—emerges solely from pre-training on next-token prediction objectives or if it is a capability that requires task-specific fine-tuning. While LLMs demonstrate generalization across tasks, the extent to which specific semantic structures are encoded in pre-trained weights versus learned during adaptation is not fully characterized. Furthermore, the relationship between model scale and the emergence of these capabilities, as well as the internal neural mechanisms (e.g., distributed vs. localized representations) supporting them, requires systematic investigation.

Methodology

The authors employ a controlled experimental design involving the training of decoder-only transformer models from scratch to isolate the effects of pre-training from fine-tuning.

Model Training: Four decoder-only transformer models (Tiny, Small, Base, Medium) ranging from 0.4M to 57M parameters were trained from scratch on the WikiText-103 corpus using a causal language modeling objective (next-token prediction). The architecture follows GPT-2/GPT-3 standards with tied embeddings and a fixed vocabulary.
Task Definition: The evaluation task is Question-Answer driven Semantic Role Labeling (QA-SRL). This reformulates SRL as extractive question answering (e.g., "Who took a walk?" to identify the Agent), avoiding explicit PropBank schema annotation during the probing phase.
Probing Protocols:
- Full Fine-tuning: All parameters (transformer and QA head) are updated to establish an upper performance bound.
- Frozen Linear Probe: The pre-trained transformer weights and positional embeddings are frozen. Only the token embeddings (to adapt to the [question] [SEP] [sentence] format) and the QA head are trained. This tests if semantic role information is decodable from pre-trained representations.
- Random Baseline: An untrained (randomly initialized) model with trained embeddings and QA head serves as a performance floor to distinguish learned representations from random capacity.
- Normalized Emergence Score: Defined as $E = \frac{F1_{frozen} - F1_{random}}{F1_{full} - F1_{random}}$ , quantifying the fraction of task performance present in pre-training.
Mechanistic Analysis:
- Temporal CKA: Centered Kernel Alignment is used to measure representational drift between fine-tuning checkpoints and the pre-trained model.
- Neuron Ablation: Principal Component Analysis (PCA) identifies "role-selective neurons" (e.g., neurons highly active for Agent or Time roles). These neurons are ablated (weights set to zero) to test causal necessity.
- Visualization: PCA and t-SNE are used to visualize the clustering of semantic roles in representation space.

Key Results

1. Emergence of Semantic Role Understanding

Pre-training Contribution: Frozen probes consistently outperform random baselines across all model scales, with a gap of 12–19 F1 points. This confirms that predicate-argument structure is encoded during pre-training alone.
Scale Dependence: While pre-training provides a consistent baseline of semantic information, the normalized emergence score decreases as model scale increases. This is because the contribution of fine-tuning ( $\Delta_{ft}$ ) grows significantly with scale (from ~8 to ~21 F1 points), while the pre-training contribution ( $\Delta_{pre}$ ) remains relatively constant. Thus, larger models rely more heavily on fine-tuning to reach peak performance, even though the foundational information is present in pre-training.
Layer-wise Encoding: Linear probes on pure pre-trained representations (without any embedding adaptation) achieve 10–26% F1, confirming that the information resides in the transformer weights. Deeper layers generally show higher decodability.

2. Internal Representation Dynamics

Representational Drift: During fine-tuning, deeper layers diverge most significantly from the pre-trained state (CKA drops to 0.25–0.49), indicating substantial reorganization of semantic representations.
Neuron Selectivity: Specific neurons in feed-forward layers exhibit selectivity for semantic roles (e.g., "Agent neurons," "Time neurons").
- Time Neurons: Show consistent causal importance across all scales; ablating them reduces performance.
- Agent Neurons: Exhibit a scale-dependent reversal. In small models, ablating Agent neurons hurts performance (causal). In larger models (Base and Medium), ablating these neurons improves performance, suggesting these neurons become interfering or redundant in larger capacities.
- Location/Manner Neurons: Show minimal causal effects, suggesting these roles are represented in a more distributed manner rather than by specific localized neurons.
Co-activation: Same-role neurons show higher co-activation correlations than cross-role neurons, but this separation is weak and increases with layer depth.

3. Replication

The findings regarding the scale-dependent transition of Agent neurons (from causal to interfering) were replicated on GPT-2 (124M, 355M) and Pythia (70M, 160M, 410M) models, confirming the phenomenon is not an artifact of the specific training setup or corpus.

Key Contributions

Systematic Methodology: A framework for distinguishing between information decodable from pre-trained weights versus information created during fine-tuning, using linear probing and normalized emergence scores.
Temporal Analysis: Introduction of a CKA-based framework to track representational evolution during fine-tuning relative to the pre-trained null model.
Neural Mechanism Identification: Discovery of role-selective neurons and evidence that semantic roles form distinct clusters in representation space, with separation increasing in deeper layers.
Scale-Dependent Reorganization: Empirical evidence that the neural implementation of semantic roles changes with scale. Specifically, the transition of Agent-selective neurons from being causally necessary in small models to being interfering in large models challenges the assumption that circuits identified in small models retain their function at larger scales.

Significance and Claims

The paper claims that semantic role understanding is a capability that emerges from pre-training alone, even at small model scales, and does not strictly require task-specific supervision to be encoded. However, the efficiency and implementation of this capability change with scale.

The authors argue that:

Emergence is not binary: It is a continuous process where pre-training provides a substantial foundation (12–19 F1 points), but fine-tuning is required to fully optimize performance, especially in larger models.
Interpretability is possible: The existence of role-selective neurons suggests that emergent linguistic capabilities can be partially localized, making them amenable to mechanistic interpretability.
Circuit Stability is not guaranteed: The finding that Agent neurons become interfering in larger models suggests that neural circuits are not static; they undergo reorganization as capacity increases. This implies that insights gained from analyzing smaller models may not directly transfer to larger, more capable systems without accounting for these structural shifts.
Cross-role interference: Larger models exhibit complex interactions where role-specific neurons may suppress other roles, a phenomenon absent in smaller models.

The work concludes that while language modeling objectives are sufficient to encode semantic role structure, the internal organization of this knowledge evolves significantly with model scale, moving from more concentrated, causal representations to distributed, and sometimes interfering, representations.