Differences in Typological Alignment in Language Models' Treatment of Differential Argument Marking

Technical Summary: Differences in Typological Alignment in Language Models' Treatment of Differential Argument Marking

Problem Statement
Recent research indicates that language models (LMs) trained on synthetic corpora can exhibit typological preferences resembling cross-linguistic regularities, particularly in structural domains like word order. However, it remains unclear whether these tendencies extend to systems where grammatical well-formedness depends on semantic factors rather than purely structural configurations. This paper investigates this gap using Differential Argument Marking (DAM), a linguistic phenomenon where arguments receive morphological marking based on semantic properties such as animacy, definiteness, and pronominality. Specifically, the authors ask whether LMs trained with standard next-token prediction objectives develop typological biases in DAM that align with human languages, and if so, whether these biases stem from the same underlying sources as human typological tendencies.

Methodology
The authors employ a controlled synthetic corpus paradigm using GPT-2-small models trained from scratch.

• Corpus Construction: Starting with a subset of the OpenSubtitles corpus (English), the authors extract approximately 21 million SVO clauses. They annotate arguments for semantic properties (animacy, definiteness, pronominality) using BERT-based classifiers.
• Rule Injection: They construct 18 distinct synthetic corpora by injecting DAM rules into the English text. These rules are parameterized along four dimensions:
  1. Semantic Trigger: Animacy, Definiteness, or Pronominality.
  2. Dependency Complexity: Local (marking depends on a single argument's property) vs. Global (marking depends on the relative prominence of subject and object).
  3. Markedness Direction: Natural (marking targets semantically atypical/less frequent arguments) vs. Inverse (marking targets typical arguments).
  4. Argument Target: Object-targeting (marking the patient) vs. Subject-targeting (marking the agent).
• Evaluation: Models are evaluated on rule mastery using minimal pairs. For each condition, 1,000 held-out sentence pairs are generated where one sentence follows the injected DAM rule and the other violates it. Performance is measured by the proportion of times the model assigns a lower negative log-likelihood (NLL) to the grammatical sentence.
• Auxiliary Analyses: The study includes tests for marker placement accuracy, semantic probing (to ensure semantic information is not lost), and BLiMP diagnostic tasks (to ensure DAM injection does not degrade general grammatical learning).

Key Results
The experiments reveal a dissociation between two major typological dimensions of DAM:

1. Markedness Direction (Strong Alignment): Models reliably reproduce the human typological preference for natural markedness. Across all semantic triggers and dependency types, models learn natural rules (where overt marking targets atypical arguments) significantly better than inverse rules.

   • Average best accuracy for Natural rules: ~85%.
   • Average best accuracy for Inverse rules: ~68%.
   • This advantage persists even when controlling for input frequency, suggesting it is driven by learnability dynamics rather than mere exposure.

2. Argument Preference (Weak/No Alignment): Models do not reproduce the strong human typological preference for object-targeting systems. While object-targeting rules showed a slight average advantage over subject-targeting rules (~79% vs. ~74%), this difference is substantially smaller than the markedness effect and statistically less significant.

   • Local rules were learned more successfully than Global rules (~77% vs. ~59%), reflecting a bias toward information locality.
   • The performance gap between natural and inverse variants was smaller for object-targeting rules than for subject-targeting rules, suggesting models encode subject semantic properties more robustly.

3. Auxiliary Findings:

   • Marker Placement: Models achieved near-ceiling accuracy in placing markers correctly, indicating that rule mastery failures are not due to an inability to locate the marker in the string.
   • Semantic Probing: Linear probes confirmed that models retain semantic information (animacy, definiteness, pronominality) for both subjects and objects, ruling out semantic degradation as a cause for performance differences.
   • BLiMP: DAM injection did not significantly impair the models' ability to learn other grammatical phenomena.

Significance and Claims
The paper argues that the observed dissociation provides evidence regarding the distinct sources of typological asymmetries:

• Markedness as Learnability: The strong alignment with natural markedness suggests that this typological tendency can emerge from distributional regularities and formal learnability. Natural markedness systems reduce prediction uncertainty (surprisal) by providing explicit local signals for less frequent configurations, aligning with the inductive biases of autoregressive models optimized for next-token prediction.
• Argument Preference as Discourse-Driven: The lack of a strong object preference in models suggests that this typological asymmetry depends on factors not captured by standard next-token training, such as discourse structure, thematic prominence, and communicative pressures (e.g., the tendency for objects to assume atypical discourse roles like topicality).
• Conclusion: Distinct typological tendencies reflect different underlying pressures. Some, like markedness direction, are accessible to LMs through local statistical learning, while others, like argument preference, likely require interactional or diachronic pressures that standard training objectives do not simulate.

Limitations
The authors note several constraints: the use of a single small architecture (GPT-2-small), the reliance on English (an SVO language where case marking is less critical than in flexible word-order languages), and the simplified operationalization of DAM triggers (treating them as independent binary features rather than interacting discourse-sensitive properties). Additionally, the frequency of marked vs. unmarked instances was not fully experimentally controlled, though the authors argue frequency alone does not explain the observed asymmetries.