Do Language Models Encode Knowledge of Linguistic Constraint Violations?

Technical Summary: Do Language Models Encode Knowledge of Linguistic Constraint Violations?

Problem Statement
While Large Language Models (LLMs) demonstrate strong performance on linguistic benchmarks and generalize grammatical regularities from raw text, the internal mechanisms enabling them to distinguish well-formed sentences from ill-formed ones remain unclear. Specifically, it is an open question whether LLMs, particularly Transformer-based architectures, encode specific "detector features" that activate upon encountering grammatical violations to penalize a sentence's log-likelihood. This study investigates the hypothesis that LLMs possess latent representations of linguistic constraint violations (denoted as the variable $G$ in the framework of Hu et al., 2026) that are selectively activated during the processing of ungrammatical inputs.

Methodology
The study employs a mechanistic interpretability approach utilizing Sparse Autoencoders (SAEs) to decompose the polysemantic activations of LLMs into sparse, monosemantic features.

• Dataset: The authors utilize the BLiMP benchmark (Warstadt et al., 2020), which provides 13 linguistic phenomena. Each phenomenon consists of minimal pairs where a well-formed sentence ($x_{good}$) and an ill-formed sentence ($x_{bad}$) differ by exactly one word. This design holds the semantic message ($m$) constant while varying the grammatical constraint ($G$).
• Models: Experiments were conducted on six models across three families: GPT-2 (small), Gemma-3 (270M, 1B, 4B), and Qwen3.5 (2B, 9B).
• Feature Decomposition: Pre-trained SAEs (from SAELens, Qwen Scope, and Gemma Scope 2) were applied to the residual streams at specific layers (25%, 50%, 65%, and 85% of model depth).
• Sensitivity Score ($\Phi$): A sensitivity score was introduced to rank SAE features based on their preferential activation on constraint-violating inputs versus well-formed inputs. Features with high positive $\Phi$ scores were selected as candidate violation detectors.
• Conjunctive Falsification Framework: To validate whether a candidate feature truly encodes a violation detector, the study proposes three jointly evaluated criteria via targeted feature ablation:
  1. F1 (Causal Effect on Violations): Ablating the candidate features must significantly increase the log-likelihood of ungrammatical sentences compared to a random control.
  2. F2 (Stability of Non-Violations): Ablation must not significantly degrade the log-likelihood of well-formed sentences (stability threshold $|\bar{M}_{good}| < 0.05$).
  3. F3 (Preferential Effect): The change in preference margin (good vs. bad) caused by ablation must be strictly positive and significantly greater than that of a random control, ensuring the effect is selective to violations.

Key Contributions

1. Sensitivity Score ($\Phi$): An unsupervised metric for identifying features preferentially activated by constraint-violating inputs, enabling the targeted selection of candidate violation features without labeled supervision.
2. Conjunctive Falsification Framework: A rigorous three-criteria test (F1, F2, F3) designed to distinguish true violation-specific detectors from features that merely correlate with general fluency or sentence likelihood.
3. Empirical Analysis: A comprehensive evaluation across 13 linguistic phenomena and six models, revealing that while models contain information related to violations, this information is not cleanly separable into unified, selective detectors under the current intervention framework.

Results
The empirical results are largely negative regarding the hypothesis of unified, selective violation detectors:

• F1 Performance: The causal effect criterion (F1) passed broadly across models and phenomena. High-$\Phi$ features are correlated with constraint-violating inputs, as their removal increases the likelihood of ungrammatical sentences.
• F2 and F3 Failures: The stability (F2) and preferential effect (F3) criteria failed in most cases. Features identified as sensitive to violations often also causally affect well-formed sentences (violating F2) or affect grammatical and ungrammatical sentences with comparable magnitude (violating F3).
• Joint Satisfaction: The joint criterion (F1 $\cap$ F2 $\cap$ F3) was satisfied by only one phenomenon (Irregular forms on GPT-2) across all models.
• Feature Entanglement: The results suggest that violation-related features are not isolated. Instead, they appear to encode general fluency information or are entangled with features representing the general message ($m$), making it difficult to isolate a pure "grammaticality" variable $G$ via ablation.

Significance and Claims
The paper concludes that current LLMs do not appear to encode a unified set of grammatical violation detectors that are selectively activated and causally distinct from general sentence representations. While the models possess information relevant to grammatical violations, this information is not cleanly separable from broader sentence-level representations under the current intervention framework.

The authors hypothesize that the negative results may stem from:

1. The sensitivity score failing to uncover features that satisfy F2 and F3.
2. SAEs failing to recover features corresponding to F2 and F3, which may be absorbed into other features.
3. The relevant features being distributed across multiple layers rather than localized to a single layer.

The study does not claim to disprove the existence of grammatical representations entirely but rather highlights the limitations of current mechanistic interpretability tools (specifically SAEs and ablation) in isolating these specific latent variables. The work is modest in its claims, noting that the main finding is negative and that alternative frameworks targeting surprisal or semantic anomalies might offer complementary perspectives.