Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

1. Problem Statement

While Transformer-based Large Language Models (LLMs) dominate computational linguistics, their validity as cognitive models of human sentence processing remains debated. Previous studies suggested Transformers could replicate human reading time patterns in "agreement attraction" phenomena (where a grammatically unrelated noun interferes with verb agreement processing). However, these prior studies suffered from four critical limitations:

1. Limited Syntactic Scope: They only tested prepositional modifier phrases, ignoring other constructions like object-extracted relative clauses.
2. Incomplete Condition Coverage: They often failed to test conditions where attraction effects should not occur (e.g., plural subjects or grammatical sentences), failing to capture the full asymmetry of human behavior.
3. Lack of Generalization: They relied on single models (typically GPT-2), making it unclear if findings were model-specific or generalizable to the Transformer architecture class.
4. Cognitive Implausibility: Some studies used bidirectional models (BERT), which are cognitively implausible for online sentence processing.

The authors aim to determine if Transformers can explain human morphosyntactic processing by systematically evaluating a diverse set of autoregressive Transformers across a comprehensive range of syntactic configurations.

2. Methodology

Experimental Design

The study utilized a surprisal-based linking mechanism, assuming a linear relationship between a token's negative log probability (surprisal) and human reading time. The authors conducted two experiments using stimuli from Wagers et al. (2009):

• Experiment 1 (Prepositional Modifiers): 192 sentences testing agreement attraction in phrases like "The slogan on the poster(s)..."
• Experiment 2 (Object-Extracted Relative Clauses): 384 sentences testing attraction in structures like "The marine(s) who the officer(s) wants/want..."

Both experiments used a 2×2×2 factorial design manipulating:

1. Subject Number: Singular vs. Plural.
2. Grammaticality: Grammatical vs. Ungrammatical (verb mismatch).
3. Attraction: Presence vs. Absence of a distractor noun with conflicting number features.

Model Selection

The authors evaluated 11 autoregressive Transformer models from four different families (GPT, GPT-2, Bloom, XGLM, Gemma-3) with parameter counts ranging from 124M to 4B.

• Constraints: Only foundation models (no instruction tuning) were used to maximize ecological validity.
• Tokenization: Surprisal values were summed for verbs split into multiple tokens to ensure accurate measurement.

Data Analysis

The authors employed three levels of analysis:

1. Pairwise t-tests: To detect attraction effects in individual models.
2. Linear Regression (Population Level): To test for two critical human-like asymmetries across the model ensemble:
   • Singular-Plural Asymmetry: Attraction should only occur with singular subjects.
   • Grammatical-Ungrammatical Asymmetry: Attraction should only occur in ungrammatical sentences.
3. Global Regression: To assess the impact of model size (parameter count) and interactions between variables.

3. Key Contributions

• Comprehensive Syntactic Evaluation: The first study to systematically compare Transformer performance on both prepositional modifiers and object-extracted relative clauses regarding agreement attraction.
• Multi-Model Ensemble Analysis: Moving beyond single-model case studies to evaluate 11 models across different families and sizes, establishing the variability within the Transformer class.
• Rigorous Asymmetry Testing: Explicitly testing for the absence of effects in conditions where humans show none, rather than just confirming the presence of effects where humans do.
• Methodological Framework: Proposing a rigorous experimental design for future cognitive evaluations of LLMs to avoid spurious generalizations.

4. Results

Experiment 1: Prepositional Modifiers

• Alignment with Humans: Transformer predictions were highly consistent with human reading time data.
• Asymmetry: Models correctly reproduced the human pattern: significant facilitatory attraction effects (faster reading) occurred only in ungrammatical sentences with singular subjects.
• Model Consistency: Predictions were stable across different model families and sizes, with high correlations in surprisal scores.
• Minor Deviation: Most models predicted a facilitatory effect for ungrammatical plural subjects (a condition where humans typically show no effect), though this is a rare configuration in human literature.

Experiment 2: Object-Extracted Relative Clauses

• Divergence: Results were chaotic and inconsistent. Models diverged significantly from human data and from each other.
• Failure to Replicate Asymmetries:
  • Models failed to reproduce the singular-plural asymmetry; many showed facilitatory effects for plural subjects where humans show inhibition or no effect.
  • Models predicted spurious facilitatory effects in grammatical sentences with plural subjects, contradicting theoretical expectations.
• Lack of Uniformity: Unlike Experiment 1, there was no consistent trend across the model ensemble. Some models (e.g., Gemma-3) deviated most dramatically, while others (e.g., GPT) were closer but still failed to capture the specific human pattern.
• Conclusion: No single model, nor the ensemble, successfully replicated the asymmetric interference patterns observed in humans for this construction.

5. Significance and Conclusion

Cognitive Adequacy: The study concludes that current Transformer models do not explain human morphosyntactic processing in a generalizable way. While they succeed in simpler prepositional configurations, they fail to capture the complex retrieval mechanisms required for object-extracted relative clauses.

Theoretical Implications:

• Architecture vs. Objective: The failure in relative clauses suggests that the next-word prediction objective or the Transformer architecture itself may lack the specific inductive biases required for human-like working memory retrieval.
• Surface Mimicry: The authors speculate that Transformers may only exhibit agreement attraction because such errors exist in their training corpora, rather than possessing a human-like cognitive mechanism for handling them.
• Monolingual vs. Multilingual: There is a tentative suggestion that monolingual models might align better with human processing than multilingual ones, though this requires further investigation.

Methodological Impact: The paper argues that evaluating LLMs as cognitive models requires comprehensive experimental designs. Relying on isolated syntactic configurations (like prepositional phrases) or single models leads to spurious generalizations. Future research must test across diverse syntactic structures and model families to determine the true cognitive limits of current AI.