Implicit Representations of Grammaticality in Language Models

Technical Summary: Implicit Representations of Grammaticality in Language Models

Problem Statement
In human language, grammaticality (conformity to syntactic rules) and likelihood (statistical frequency in natural text) are distinct concepts. A sentence can be grammatical but highly unlikely (complex meaning), or ungrammatical but likely (common error). Pretrained Language Models (LMs) are probabilistic systems trained to maximize corpus likelihood. While they perform well on minimal-pair tasks (distinguishing a grammatical sentence from a minimally altered ungrammatical one), their raw string probabilities often fail to sharply discriminate between grammatical and ungrammatical sentences in general, as they conflate syntactic correctness with semantic plausibility and frequency.

The central question addressed is whether LMs, despite being trained solely on likelihood, implicitly acquire a distinct representation of grammaticality within their hidden layers that is separable from string probability.

Methodology
The authors propose a linear probing approach to investigate the internal representations of LMs without relying on costly human-annotated grammaticality datasets for training.

1. Synthetic Training Data Generation:

   • Source: 50,000 sentences were sampled from the Penn Treebank and Project Gutenberg to serve as "grammatical" examples.
   • Perturbation: Ungrammatical examples were generated by applying three noise functions to the original sentences:
     • Insertion: Randomly inserting 1–5 tokens from the corpus.
     • Deletion: Randomly removing 1–5 tokens.
     • Local Shuffle: Randomly permuting a contiguous 5-token window.
   • Validation: An LLM judge confirmed that 93.72% of the perturbed sentences were indeed ungrammatical.
   • Split: 80% of the data was used for training, and 20% for development (hyperparameter tuning).

2. Probe Architecture:

   • A logistic regression classifier (linear probe) was trained on the hidden states of the LM at the last token of a sentence.
   • The probe was trained to predict the binary label (grammatical vs. ungrammatical).
   • Experiments utilized $\ell_2$ regularization (standard) and $\ell_1$ regularization (LASSO) to identify sparse subsets of neurons responsible for the signal.

3. Evaluation Framework:

   • Models: Tested on open-weight base models (OLMo-2/3, Llama-3.2/3.1, Gemma-2) excluding instruction-tuned variants to avoid alignment confounds.
   • Benchmarks:
     • Grammaticality: BLiMP (minimal pairs), CoLA, and SyntaxGym (standalone acceptability).
     • Cross-lingual: Six languages (Swedish, Dutch, Italian, Russian, Japanese, Chinese) using zero-shot transfer.
     • Semantic Plausibility: Three datasets where both minimal-pair members are grammatical but differ in semantic plausibility.
   • Metrics: Accuracy (ACC) for minimal pairs and Area Under the ROC Curve (AUC) for standalone acceptability.
   • Baselines: Comparison against LM string log-probabilities (length-normalized cumulative logprobs).

Key Results

• Superiority over String Probabilities: Probes trained on the synthetic dataset significantly outperformed LM log-probabilities on grammaticality benchmarks (BLiMP, CoLA, SyntaxGym). This held true for both minimal-pair and non-minimal-pair settings.
• Distinctness from Likelihood:
  • The probe scores showed only a moderate correlation (Spearman's $\rho \approx 0.23–0.47$) with LM log-probabilities, suggesting they capture distinct information.
  • Adding log-probabilities as an extra feature to the probe did not consistently improve performance, implying the hidden states already encode the necessary likelihood information relevant to grammaticality.
  • Conversely, when predicting log-probabilities from hidden states, ridge regression achieved moderately high $R^2$ ($0.51–0.73$), confirming that likelihood information is present in the representations.
• Semantic Selectivity: On semantic plausibility benchmarks, the grammaticality probe performed worse than string probabilities. This indicates the probe is selectively tuned to syntactic acceptability rather than semantic likelihood, effectively disentangling the two concepts.
• Cross-Lingual Generalization: A probe trained exclusively on English data generalized to typologically diverse languages (Swedish, Dutch, Italian, Russian, Japanese, Chinese), often outperforming the native-language string probabilities of the models.
• Neuron Localization: Using LASSO, the authors found that grammaticality signals could be captured by extremely small subsets of neurons (as few as 0.01% of the total, roughly 10 neurons), though the signal is also distributed across layers.

Significance and Claims
The paper argues that while LMs are fundamentally statistical systems trained on likelihood, their large-scale training induces an implicit grammaticality distinction within their hidden representations.

• Challenge to Skepticism: The results challenge the view (e.g., Katzir, 2023) that the grammaticality–likelihood distinction is "entirely foreign" to LMs. The authors suggest that LMs do acquire a notion of grammaticality distinct from output probabilities, even without explicit linguistic supervision.
• Methodological Contribution: The study demonstrates that synthetic perturbations of natural text can serve as a robust, scalable proxy for training grammaticality probes, bypassing the need for small, expensive human-curated datasets for training (though those are still used for evaluation).
• Implications for Linguistic Cognition: While the authors refrain from claiming LMs are complete models of human linguistic acquisition, they posit that LMs serve as a viable testbed for studying how syntactic knowledge can emerge from distributional learning. The findings suggest that the "competence–performance" gap observed in humans (where a speaker knows a rule but may produce errors) has a parallel in LMs, where hidden representations encode grammaticality distinct from the noisy likelihood of surface strings.

Limitations Acknowledged by the Authors

• The synthetic training data is imperfect; original sentences may contain errors, and perturbations may introduce semantic implausibility rather than pure syntactic errors.
• The study focuses on natural languages and does not address formal languages or the learning dynamics of when grammaticality is acquired during training.
• The work examines static representations at the final checkpoint, not the evolution of these representations over training time.