Large Language Models as Proxies for Theories of Human Linguistic Cognition

Technical Summary: Large Language Models as Proxies for Theories of Human Linguistic Cognition

Problem Statement
The paper addresses the role of Large Language Models (LLMs) in the study of Human Linguistic Cognition (HLC). It critiques the "LLM Theory" (e.g., Piantadosi 2023), which posits that current LLMs are themselves the best theories of HLC, arguing that this view fails to account for fundamental distinctions in human cognition, such as competence vs. performance, grammaticality vs. likelihood, and specific inductive leaps regarding hierarchical structure.

Instead, the paper investigates the "Proxy View." This hypothesis suggests that while current LLMs are not good theories of HLC themselves, they may serve as proxies for a third, as-yet-undisclosed theory ($H_3$). $H_3$ is envisioned as a theory that is relatively linguistically neutral (lacking the strong innate biases of generative linguistics, $H_1$) yet capable of explaining human linguistic acquisition and typological patterns. The central problem is that $H_3$ is currently unspecified, making it difficult to evaluate empirically. The authors ask whether current LLMs can function as methodological tools to test whether a linguistically-neutral $H_3$ could successfully acquire linguistic knowledge from realistic corpora or explain cross-linguistic typological asymmetries.

Methodology
The authors evaluate the Proxy View through two primary experimental frameworks, using a suite of eight LLMs ranging from small models trained on child-directed speech (CHILDES, BabyLM) to massive models trained on billions of tokens (BERT, GPT-2, LLaMA-3.2).

1. Alignment with the Stimulus (Acquisition of Inductive Leaps):

   • Task: The authors test whether LLMs can acquire specific linguistic generalizations that are argued to be difficult for linguistically-neutral learners to derive from the "poverty of the stimulus" (APS).
   • Phenomena: Three specific syntactic phenomena were tested:
     • Across-the-Board (ATB) movement.
     • Parasitic Gaps (PG).
     • That-trace effects (TTE).
   • Metric: Using a "minimal pair" methodology, the models were presented with grammatical and ungrammatical sentence pairs. Success was defined leniently: the model assigns a higher probability to the grammatical continuation than the ungrammatical one at a critical token region.
   • Logic: If LLMs (as proxies for $H_3$) fail to prefer the grammatical option even after training on developmentally realistic or massive corpora, it suggests that a linguistically-neutral $H_3$ would also fail to acquire these patterns, thereby strengthening the APS argument for innate biases ($H_1$).

2. Cross-Linguistic Evidence (Typological Asymmetries):

   • Task: The authors examine whether LLMs learn "attested" languages more easily than "unattested" (perturbed) languages. If a linguistically-neutral $H_3$ can explain why certain language types are unattested, it should be harder for the proxy LLM to learn the unattested variants.
   • Setup: Building on Kallini et al. (2024), the authors trained a Transformer on corpora of English, Italian, Russian, and Hebrew, as well as on perturbed versions of these corpora.
   • Perturbations:
     • Partial-reverse: Reversing sentence segments from a random index.
     • Full-reverse: Reversing entire sentences.
     • Switch-indices: Swapping tokens at specific indices (0 and 2).
     • Token-hop: Inserting markers at fixed linear distances from verbs (requiring linear counting).
   • Metric: Validation perplexity during training. Lower perplexity indicates easier learning.

Key Results

• Acquisition of Syntactic Generalizations:

  • The LLMs generally failed to show a preference for grammatical over ungrammatical sentences in the minimal pairs for ATB, PG, and TTE.
  • With the exception of the massive LLaMA model (trained on data equivalent to hundreds of thousands of years of human exposure), all models overwhelmingly preferred the ungrammatical options.
  • This failure suggests that if LLMs are proxies for $H_3$, then $H_3$ would likely fail to acquire these complex hierarchical dependencies from realistic input, failing to undermine the APS arguments for innate linguistic biases.

• Typological Asymmetries:

  • The results did not consistently support the idea that attested languages are easier to learn than unattested perturbations.
  • While partial-reverse perturbations were harder to learn than baselines (supporting the Proxy View for this specific case), other perturbations yielded contradictory results.
  • For full-reverse, switch-indices, and token-hop, the models often found the unattested (perturbed) versions easier to learn than the attested baselines, or found no significant difference.
  • Specifically, the "token-hop" perturbation (which violates hierarchical generalizations in favor of linear counting) was sometimes learned more easily than the attested baseline.

Key Contributions

• Methodological Clarification: The paper articulates the "Proxy View" as a distinct position from the "LLM Theory," clarifying that LLMs could theoretically serve as tools to evaluate neutral theories without being those theories themselves.
• Empirical Evaluation: It provides a preliminary empirical test of the Proxy View, moving beyond theoretical speculation to concrete experiments on acquisition and typology.
• Negative Evidence: The study demonstrates that, under current conditions and with current models, LLMs do not provide evidence that a linguistically-neutral theory can account for key linguistic phenomena or typological patterns.

Significance and Claims
The paper concludes that, at present, current LLMs do not support the Proxy View. The models' inability to approximate human-like inductive leaps (even with massive data) and their inconsistent learning of typological patterns suggest they are poor proxies for a successful linguistically-neutral theory of HLC.

The authors emphasize that the Proxy View remains "too vague" to be directly useful in empirical science because the target theory ($H_3$) is not explicitly defined. The paper serves as a "plea" for proponents of the Proxy View to provide explicit details of the target theory and to demonstrate how LLM performance specifically supports that theory over generative alternatives. The authors maintain a modest stance, noting that their negative results are specific to the models and phenomena tested, and that future work with different models or more explicit theories could yield different conclusions. However, based on current evidence, LLMs do not suggest a need to change existing linguistic theory.