Do Language Models Know What Not to Say? Causal Evidence for Statistical Preemption in LLMs

Technical Summary: Causal Evidence for Statistical Preemption in LLMs

Problem Statement
The paper addresses "Baker's Paradox," a fundamental puzzle in language acquisition: how learners acquire knowledge of what is unacceptable (negative evidence) without ever being explicitly told that certain sentences are wrong. In the context of Construction Grammar, two competing hypotheses attempt to explain this "retreat from overgeneralization":

1. Statistical Preemption: Learners acquire negative knowledge by tracking the frequency of specific, conventional competing forms (e.g., hearing "donated the books to the library" frequently preempts the unattested "*donated the library the books").
2. Entrenchment: Learners resist novel constructions simply because the verb is frequently encountered in any construction, reducing willingness to use it in new frames.

While behavioral studies have debated these mechanisms, the paper posits that Large Language Models (LLMs), trained purely on distributional statistics without explicit grammatical instruction, offer a controlled environment to test whether distributional learning alone suffices to acquire this negative knowledge.

Methodology
The authors conducted four experiments spanning 120 English verb–construction pairings across three alternation types: dative, causative, and locative. They evaluated 14 models from four families (GPT-2, Pythia, LLaMA-2, OLMo) ranging from 124M to 70B parameters.

• Linking Hypothesis: The study uses word-level surprisal ($S(w_t) = -\log_2 P(w_t | w_{<t})$) as a proxy for human acceptability. The key metric is the surprisal differential ($\Delta S$) between an "unconventional" (preempted) form and a "conventional" form.
• Experiment 1 (Correlational): Tested if LLM $\Delta S$ patterns correlate with human acceptability judgments from three independent datasets (DAIS, Robenalt & Goldberg, Tachihara & Goldberg).
• Experiment 2 (Dissociation): Used a +Competing/−Competing design (adapted from Winkler et al., 2015) to dissociate preemption from entrenchment. Verbs were matched on overall frequency but differed in whether a specific competing conventional form existed. The study employed non-circular partial correlations against human data to ensure the results were not artifacts of shared corpus statistics.
• Experiment 3 (Scaling): Analyzed preemption sensitivity across model sizes (160M to 12B parameters in Pythia) to determine if the effect scales as a power law.
• Experiment 4 (Causal Intervention): Conducted controlled fine-tuning on GPT-2 124M with manipulated frequencies. Three conditions were tested:
  • Amplified: Increased frequency of the conventional form.
  • Attenuated: Equalized frequencies of both forms.
  • Reverse: Increased frequency of the unconventional form (a control to rule out simple frequency sensitivity).
  • The experiment was replicated across five random seeds.

Key Results

1. Human-LLM Alignment: LLM surprisal patterns strongly correlate with human acceptability judgments ($r = 0.79$ for LLaMA-2 7B against the DAIS benchmark), validated across three independent behavioral datasets.
2. Preemption vs. Entrenchment: The observed effects are driven by the frequency of the competing form, not the overall verb frequency. In the +Competing/−Competing design, verbs with a competing conventional form showed significantly higher $\Delta S$ than frequency-matched verbs without such a competitor. Partial correlations confirmed that preemption explains variance in human data ($r_{partial} = 0.58$) while entrenchment does not.
3. Scaling: Preemption sensitivity scales as a power law with model size ($r(N) = a \cdot N^b + c$), with an exponent $b \approx 0.092$, indicating diminishing returns but no sudden phase transitions.
4. Causal Evidence: Fine-tuning interventions causally shifted preemption behavior. Increasing the conventional form's frequency strengthened preemption ($\Delta \Delta S = +0.73$), while increasing the unconventional form's frequency had a significantly weaker effect ($\Delta \Delta S = -0.29$). This asymmetry supports the theory that preemption operates through the inhibitory force of established alternatives, rather than simple frequency sensitivity.

Significance and Claims
The paper claims to provide the first computational study to directly dissociate statistical preemption from entrenchment within a single converging design using LLMs. Its primary contributions are:

• Mechanistic Validation: It demonstrates that neural language models acquire negative linguistic knowledge through distributional competition, supporting the core mechanism of Construction Grammar.
• Causal Proof: Through controlled fine-tuning and reverse-direction controls, it offers causal evidence that manipulating competing-form frequencies shifts model behavior in the direction predicted by preemption theory.
• Methodological Rigor: It addresses circularity concerns in prior work by using non-circular partial correlations against human data and employing a reverse-direction control to rule out tautological frequency-sensitivity explanations.
• Generalizability: The findings hold across multiple construction types (dative, causative, locative) and model architectures.

The authors position their work as a complement to existing causal studies (e.g., Yao et al., 2025; Misra and Mahowald, 2024), noting that while they do not claim unique priority for "causal" evidence, their specific combination of preemption-entrenchment dissociation, non-circular validation, and reverse-direction asymmetry controls provides a robust, converging line of evidence. The study concludes that for English, Baker's Paradox has an answer in distributional learning: models develop preemption sensitivity shaped by the frequency of conventional competitors, mirroring human acquisition.