Quantifying the cross-linguistic effects of syncretism on agreement attraction

Technical Summary: Quantifying the cross-linguistic effects of syncretism on agreement attraction

Problem Statement
Agreement attraction occurs when a verb erroneously agrees with an intervening noun (the attractor) rather than its grammatical head. Psycholinguistic research, particularly within the cue-based retrieval framework (Wagers et al., 2009; Dillon et al., 2013), suggests that morphological syncretism—where distinct syntactic features share the same overt morphology—modulates these errors. In languages like English, German, and Russian, syncretic attractors (e.g., accusative forms identical to nominative) increase attraction rates compared to non-syncretic ones. However, this pattern is not universal; Turkish and Armenian show no such modulation, and French exhibits a reverse pattern. Currently, there is no principled account explaining why syncretism affects agreement attraction differently across languages.

Methodology
The authors utilize Large Language Models (LLMs) as a cross-linguistically uniform processing proxy to investigate this variation across four languages: English, German, Russian, and Turkish. They extract two primary measures from LLMs to serve as proxies for human processing difficulty and memory retrieval competition:

1. Surprisal: Calculated using autoregressive models (GPT-2 style) to measure the negative log probability of the verb given its context. Lower surprisal in ungrammatical sentences indicates the model is "led astray" by the attractor, mirroring human attraction errors.
2. Attention Entropy: Calculated using bidirectional models (BERT-style) to measure the dispersion of attention from the verb position to potential agreement controllers. Higher entropy toward an intervening attractor suggests greater retrieval competition.

To ensure syntactic relevance, the authors employed probing techniques to identify specific attention heads and layers that best track subject-verb dependencies for each language, rather than aggregating across all layers. They fit Bayesian mixed-effects models to compare conditions varying by Grammaticality (grammatical vs. ungrammatical), Attractor Number (singular vs. plural), and Syncretism (syncretic vs. non-syncretic). The key metric is the interaction between these factors, specifically whether the attraction effect (difference between plural and singular attractors) is modulated by syncretism.

Key Results
The study replicates behavioral findings in three of the four languages, with varying degrees of success:

• English and German: The LLM-derived measures successfully replicate the behavioral finding that syncretism amplifies agreement attraction. In both languages, the difference in surprisal and attention entropy between plural and singular attractors is significantly larger in syncretic conditions than in non-syncretic ones.
• Turkish: The results align with behavioral data showing no modulation by syncretism. The LLMs show a general attraction effect (plural attractors reduce surprisal), but this effect is not significantly different between syncretic and non-syncretic conditions.
• Russian: The findings are mixed. The LLMs capture the standard attraction effect in accusative (syncretic) conditions. However, they fail to fully reproduce the behavioral "reversed" attraction effect observed in genitive conditions, where singular attractors (syncretic with nominative plural) cause more interference than plural attractors. The LLMs show a much weaker or non-existent reversal, suggesting they rely more heavily on syntactic context to disambiguate forms that humans find morphologically similar.

Key Contributions

• Cross-linguistic Quantification: The paper provides the first computational quantification of syncretism effects on agreement attraction across four typologically diverse languages using a unified methodology.
• Validation of LLMs as Proxies: It demonstrates that LLM-derived surprisal and attention entropy can serve as valid proxies for human agreement attraction phenomena, successfully replicating the presence or absence of syncretism modulation in English, German, and Turkish.
• Identification of Model Limitations: The study highlights a specific divergence in Russian, where LLMs appear less susceptible to "spurious" morphological overlaps (pseudo-plurals) than human comprehenders, suggesting a difference in how humans and models weight morphological cues versus syntactic structure.

Significance and Claims
The authors claim that this work establishes a baseline for future research into how morphological encoding influences memory retrieval. They argue that while LLMs are sensitive to the same syncretisms that drive human attraction errors, the underlying mechanisms may differ. Specifically, the results suggest that languages differ systematically in how much weight they place on morphological cues versus other factors (like word order or syntax) to establish agreement dependencies. The paper posits that syncretism effects are strongest in languages where morphological distinctions are few but highly informative (e.g., English, German) or where case encodes dense feature bundles (e.g., Russian), whereas in agglutinative languages with transparent morphology (e.g., Turkish), both speakers and models rely less on specific syncretic overlaps, resulting in null effects. The authors explicitly state that their findings do not yet provide a complete principled account for the cross-linguistic variation but offer a computational framework to further investigate these dependencies.