Beyond next-word prediction: hierarchical linguistic composition drives LLM-brain alignment in time

By manipulating linguistic features in predictability-matched sentences, this study demonstrates that hierarchical linguistic composition, particularly syntactic structure and associative semantics, significantly drives the alignment between LLM representations and human brain activity, while compositional semantics appears to be more uniquely encoded in the human brain.

The Gist

Imagine your brain and a super-smart computer (a Large Language Model, or LLM) are both listening to a story being read aloud. Scientists have noticed that when humans listen, their brain waves "dance" in a similar rhythm to the internal calculations the computer makes. But why? Is it just because both the brain and the computer are good at guessing what word comes next (like finishing a sentence), or is there something deeper about how they understand the structure of language?

This paper treats the brain and the computer like two different chefs trying to recreate the same dish. The researchers wanted to know: Are they just following the same recipe because they both know the ingredients well (statistical patterns), or do they actually understand the cooking process (hierarchical composition) in the same way?

To find out, the team served up sentences to both human volunteers (while recording their brain waves with an EEG cap) and a computer model called GPT-2. They carefully tweaked the sentences in three specific ways, making sure that the "guessability" of the words remained the same so that the only difference was the type of meaning:

1. The "Grammar Skeleton" (Syntactic Structure): They looked at sentences with clear, organized grammar.

   • The Result: When the sentences had strong grammatical structure, the computer's internal "thoughts" and the human brain waves matched up even better. It's like finding that both the chef and the computer are using the same specific knife skills when chopping vegetables.

2. The "Building Block Meaning" (Compositional Semantics): This is when the meaning of a phrase is built strictly from the meanings of its parts (like "red car" meaning a car that is red).

   • The Result: Surprisingly, when the sentences relied heavily on this kind of building-block meaning, the match between the computer and the brain dropped. It's as if the human chef started using a secret family technique that the computer simply didn't have. The human brain seems to handle this specific type of meaning in a unique way that the computer doesn't quite replicate.

3. The "Word Association" (Associative Semantics): This is when words are linked by loose connections or habits (like thinking of "butter" when you hear "bread").

   • The Result: Changing these associations didn't change the match at all. The computer and the brain were already on the exact same page regarding these loose connections. It's like both chefs automatically knowing that "salt" goes with "pepper" without needing a special instruction.

The Big Takeaway
The study shows that the connection between human brains and AI isn't just about predicting the next word. The computer and the brain are actually "speaking the same language" when it comes to grammar and word associations. However, when it comes to building complex meanings from smaller parts, the human brain has a special, unique way of doing it that the computer hasn't quite mastered yet. The computer is a great mimic of our habits and rules, but our brains have a unique flair for constructing meaning that the machine is still learning to copy.

Technical Summary

Technical Summary: Beyond Next-Word Prediction

Problem Statement
The internal representations of Large Language Models (LLMs) have been observed to correlate with human neural activity during language comprehension. However, the underlying mechanism driving this "LLM-brain alignment" remains debated. One prevailing hypothesis suggests this alignment stems merely from shared sensitivity to statistical patterns (e.g., predictability) inherent in both systems. An alternative view posits that the alignment reflects the emergence of shared, structured linguistic representations. This study investigates whether hierarchical linguistic composition—a property considered fundamental to human language—is the specific driver of this alignment, distinct from mere statistical predictability.

Methodology
To disentangle linguistic structure from statistical factors, the authors employed a controlled experimental design involving both an LLM (GPT2-XL) and human participants:

• Stimuli Manipulation: English sentences were systematically manipulated across three linguistic dimensions: syntax, compositional semantics, and associative semantics.
• Predictability Matching: Crucially, the linguistically manipulated stimuli were matched for predictability. This control allows the researchers to isolate the effects of linguistic structure from the effects of statistical probability.
• Data Collection: Human participants underwent an electroencephalography (EEG) experiment while processing these sentences. Simultaneously, contextual embeddings were extracted word-by-word from the hidden layers of GPT2-XL.
• Analysis: The study utilized a linear encoding model to compute LLM-EEG alignment scores. By comparing these scores across predictability-matched conditions, the authors evaluated how specific linguistic manipulations modulate the alignment between human brain activity and model representations.

Key Contributions and Results
The study identifies three distinct patterns regarding how linguistic features modulate LLM-brain alignment, demonstrating that these effects occur above and beyond predictability:

1. Syntactic Structure: Alignment increases for word sequences possessing syntactic structure, suggesting that both the LLM and the human brain encode syntactic hierarchy in a comparable manner.
2. Compositional Semantics: Alignment decreases for sentences involving compositional semantics. This indicates that the encoding of compositional meaning is more unique to the human brain and less shared with the LLM's internal representations.
3. Associative Semantics: This dimension does not significantly modulate alignment, implying that associative semantic information is encoded similarly by both the LLM and the human brain.

Significance
The paper claims that hierarchical linguistic composition is a critical factor driving LLM-brain alignment, challenging the view that alignment is solely a byproduct of statistical pattern matching. The results suggest a nuanced divergence in how these systems process language: while LLMs and the human brain share representations for associative semantics and certain aspects of syntactic structure, the human brain possesses a distinct capacity for encoding compositional semantics that is not fully mirrored in the GPT2-XL model. This work provides empirical evidence that LLM-brain alignment is driven by specific linguistic properties rather than general statistical co-occurrence.