Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

1. Problem Statement

Large Language Models (LLMs) exhibit a puzzling heterogeneity in their formal linguistic competence. While they achieve near-perfect mastery on many linguistic phenomena, they often perform significantly below chance (random guessing) on specific syntactic paradigms, even after training on trillions of tokens.

The central research question is whether these failures stem from:

1. Inherent Architectural Limitations: The model lacks the necessary inductive biases or representational capacity to learn specific grammatical rules.
2. Data Scarcity: The specific grammatical constructions are underrepresented in standard web-scale corpora (e.g., FineWeb), leading to insufficient exposure during pre-training.

Previous scaling laws have shown that simply increasing the number of pre-training tokens does not reliably resolve these specific failures; in some cases (e.g., principle_A_reconstruction), performance follows an inverse U-shaped curve, degrading as training continues.

2. Methodology

The authors propose a controlled data intervention strategy to isolate the impact of data scarcity from architectural constraints.

• Base Model: GPT-2 Small (124M parameters).
• Pre-training Budget: 100M tokens (sampled from the FineWeb corpus). This budget is comparable to the linguistic input a 13-year-old child receives in a lifetime, contrasting with the trillions of tokens used by modern LLMs.
• Intervention Strategy:
  • Synthetic Data Generation: Using GPT-OSS 120B, the authors generated a synthetic corpus containing sentences that instantiate specific BLiMP (Benchmark of Linguistic Minimal Pairs) paradigms.
  • Diversity Control: To ensure the synthetic data did not distort the model's general distribution, the generation process included a curated taxonomy of genres (e.g., legal, medical, fiction) and a pool of 10,000 frequent lemmas. The LLM was instructed to write naturalistic documents in specific genres containing the target syntactic structure.
  • Injection: 1% of the pre-training data (1M tokens, containing ~3,000 target sentences) was replaced with this targeted synthetic data.
• Experimental Conditions:
  1. Baseline: 100M tokens from FineWeb only.
  2. Random Control: 1% random synthetic text (no specific linguistic target).
  3. Targeted Intervention: 1% synthetic text specifically targeting one of the 9 worst-performing BLiMP paradigms.
• Evaluation: Models were evaluated on the full BLiMP suite (67 paradigms). The metric is accuracy in assigning higher probability to the grammatical sentence over its minimally different ungrammatical counterpart.

3. Key Contributions

1. Identification of Failure Modes: The study identifies 9 specific BLiMP paradigms where standard GPT-2 models perform below chance (e.g., only_npi_scope, principle_A_reconstruction).
2. Causal Evidence for Data Scarcity: By injecting a minimal amount (1%) of targeted data, the authors provide an "existence proof" that these failures are primarily due to data scarcity rather than architectural limitations.
3. Efficiency of Targeted Intervention: The study demonstrates that massive scaling is not required to fix specific linguistic deficits; a tiny, high-quality injection of data is sufficient to recover performance.
4. Preservation of General Competence: The interventions improved specific paradigms without causing catastrophic forgetting or degrading aggregate performance.

4. Key Results

A. Paradigm-Specific Improvements

Targeted data injection led to substantial accuracy gains in 8 out of 9 targeted paradigms:

• only_npi_scope: Accuracy surged from 20.9% (Baseline) to 69.4% (+48.5 points).
• principle_A_reconstruction: Improved from 37.2% to 78.9% (+41.7 points).
• existential_there_quantifiers_2: Improved from 23.4% to 51.8% (+28.4 points).
• Other paradigms (wh_vs_that_with_gap_long_distance, left_branch_island_echo_question, etc.) showed gains ranging from +11.8 to +17.9 points.

B. The Resistant Case

• principle_A_c_command: This paradigm remained below chance (Baseline: 49.0%, Targeted: 46.0%) even after intervention. The authors hypothesize this may require significantly higher data density or stronger inductive biases due to complex long-range dependencies combined with linearly close distractors.

C. Ablation Studies (Data Magnitude)

• The study tested intervention magnitudes of 0.01%, 0.1%, and 1%.
• Non-monotonicity: While most paradigms improved with more data, some (like only_npi_scope) showed non-monotonic behavior, suggesting complex interactions between synthetic data and existing corpus statistics.
• Low Data Requirement: For principle_A_reconstruction, even a 0.01% injection (approx. 30 examples) was sufficient to boost accuracy from 37.2% to 72.3%, indicating the model needs very little evidence to learn these structures if exposed.

D. Aggregate Performance

• Contrary to fears of "catastrophic forgetting," the targeted interventions generally preserved or slightly improved the aggregate BLiMP score (e.g., moving from 74.77% to ~76.4% in some configurations).
• The Random Control model performed similarly to the Baseline, confirming that improvements were due to the specific syntactic evidence, not just the presence of synthetic text.

E. Comparison to Scaled Models

• A 124M parameter model with targeted intervention on principle_A_reconstruction (78.9%) significantly outperformed a Llama-3 70B model trained on 15T+ tokens (49.6%) on the same paradigm. This suggests data composition is more critical than data scale for acquiring formal linguistic competence.

5. Significance and Implications

• Reframing the Bottleneck: The paper challenges the prevailing "scale-only" paradigm, arguing that for formal linguistic competence, the composition of pre-training data is the primary bottleneck, not the model architecture or total token count.
• Human-Scale Modeling: The findings support the "BabyLM" challenge and the feasibility of training small models on human-scale data budgets (e.g., 100M tokens) to achieve robust grammatical knowledge, provided the data is curated to include sufficient distributional evidence of rare constructions.
• Future Directions: The work suggests that future efforts in language modeling should focus on data curation and synthetic augmentation to target specific linguistic gaps, rather than indiscriminately scaling up token counts. It also highlights that some complex syntactic phenomena (like principle_A_c_command) may remain resistant to simple data injection, pointing to areas where architectural inductive biases might still be necessary.

In conclusion, the paper provides strong empirical evidence that the "puzzling disparity" in LLM linguistic competence is largely a data problem, solvable through targeted, minimal data interventions.