Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?

1. Problem Statement

The paper addresses a central debate in computational linguistics and cognitive science: Can domain-general language models (LMs) learn abstract, shared syntactic representations for filler-gap dependencies without language-specific inductive biases?

• The Phenomenon: Filler-gap dependencies occur when a constituent (the "filler") is displaced from its canonical position (the "gap"), such as in wh-questions ("What did the student make _?") or topicalization ("This robot, the student made _").
• The Gap in Literature: Previous causal interpretability studies (e.g., Boguraev et al., 2025) using massive models trained on web-scale data found evidence of shared representations across different constructions. However, these models are trained on data volumes far exceeding human exposure, raising questions about whether these findings apply to human language acquisition.
• The Core Question: Do LMs trained on developmentally feasible data (mimicking child input) develop shared representations for filler-gap dependencies, or do they rely on item-specific, construction-specific mechanisms that require superhuman amounts of data to generalize?

2. Methodology

2.1 Model and Data

• Model: The study utilizes the BabyLM-100M model (based on GPT-2-small architecture).
• Training Data: The model was trained on the BabyLM Strict-100M corpus, containing ~100 million tokens designed to mimic the linguistic input of an English-speaking child up to age 12.
• Checkpoints: The authors evaluated 19 checkpoints ranging from 1M to 100M tokens to track the developmental trajectory of learning.

2.2 Experimental Design

The study focuses on two syntactic constructions with vastly different input frequencies:

1. High-Frequency Source: Wh-questions (common in child-directed speech).
2. Low-Frequency Sink: Topicalization (rare in child-directed speech).

The experiments utilize Distributed Alignment Search (DAS), a causal intervention method from mechanistic interpretability (Geiger et al., 2024).

• Mechanism: DAS learns a 1-dimensional vector direction ($a$) in the model's representation space that aligns with a binary variable: FILLER_PRESENT (0 or 1).
• Intervention: The difference between the internal representation of a "source" sentence (with a filler) and a "base" sentence (without a filler) is projected onto this learned vector and injected into the base sentence.
• Goal: If the intervention successfully shifts the model's next-token prediction from the base label to the source label, it confirms the existence of a causal, shared representation for the dependency.

2.3 Experimental Conditions

The authors tested four conditions to evaluate localization and transfer:

1. Wh $\to$ Wh: Within-construction localization (High freq $\to$ High freq).
2. Topic $\to$ Topic: Within-construction localization (Low freq $\to$ Low freq).
3. Wh $\to$ Topic: Forward transfer (High freq $\to$ Low freq).
4. Topic $\to$ Wh: Backward transfer (Low freq $\to$ High freq).

They also controlled for animacy (animate vs. inanimate fillers) to test for "lexical boost" effects.

2.4 Evaluation Metric

The primary metric is ODDS, which quantifies the magnitude of the causal effect by measuring the shift in log-probabilities toward the counterfactual outcome.

• Thresholds: Values near 0 indicate no effect; 3–6 indicate emerging structure; >8 indicate strong causal mechanisms.

3. Key Results

3.1 Developmental Trajectory (RQ1)

• Delayed Emergence: Causal effects (MAX ODDS) increased monotonically with training tokens but remained weak until late in training.
• Misalignment with Human Development: Strong causal effects (ODDS > 8) only emerged after 50M–100M tokens.
• Implication: Humans acquire sensitivity to filler-gap dependencies by ~18 months (approx. 1M–2M tokens of input). The model failed to show robust generalization at the 10M token checkpoint (equivalent to ages 2–5), supporting the Misaligned Emergence Hypothesis.

3.2 Construction Specificity (RQ2)

• Localization > Transfer: Within-construction localization (Wh $\to$ Wh, Topic $\to$ Topic) consistently outperformed cross-construction transfer.
• Quantitative Gap: At 100M tokens, within-construction effects were significantly higher (Mean difference = 1.33 ODDS) than cross-construction effects.
• Conclusion: The model learns item-sensitive and construction-specific representations rather than a single abstract mechanism.

3.3 Transfer Asymmetry (RQ3)

• Counter-Intuitive Transfer: Contrary to the "Frequency Modulation Hypothesis" (which predicted High $\to$ Low transfer would be stronger), the study found Topic $\to$ Wh transfer was significantly stronger than Wh $\to$ Topic.
• Explanation: The authors suggest that because topicalization is so rare in the input, the model may learn a more general mechanism for it when it does appear, whereas wh-questions are learned as highly specific, frequent patterns that do not generalize well to rarer structures.
• Note: This asymmetry diminished in models trained on even larger datasets (1B+ tokens), suggesting the effect is specific to the developmental constraints of the BabyLM scale.

3.4 Lexical Boost

• The study replicated the "lexical boost" effect: Transfer was significantly stronger when the animacy of the training and evaluation items matched (e.g., animate $\to$ animate), confirming that semantic features influence syntactic representation learning.

4. Key Contributions

1. Developmental Validity: This is one of the first studies to apply causal interpretability (DAS) to BabyLM-scale models, bridging the gap between large-scale LM analysis and human language acquisition theories.
2. Evidence for Construction Specificity: It provides causal evidence that LMs trained on realistic data do not immediately form abstract, shared representations for filler-gap dependencies; instead, they rely on construction-specific mechanisms.
3. Refutation of "Domain-General Sufficiency": The results challenge the notion that domain-general statistical learning alone is sufficient for human-like syntactic acquisition. The model required superhuman amounts of data (100M tokens) to achieve generalizations that children make with significantly less input.
4. Methodological Extension: Successfully applied DAS to track the emergence of syntactic representations over a developmental timeline, rather than just analyzing a static, fully trained model.

5. Significance and Implications

• For Language Acquisition: The findings support the hypothesis that human learners require language-specific inductive biases (e.g., innate constraints or structured hypothesis spaces) to learn syntactic generalizations efficiently. Pure next-word prediction models, even with child-like data, fail to replicate the speed and nature of human acquisition.
• For AI Safety and Alignment: It highlights that current LMs may possess "brittle" syntactic knowledge that is highly sensitive to construction type and training frequency, rather than robust, abstract grammatical rules.
• Future Directions: The authors argue for modeling language acquisition with explicit architectural biases and evaluating models not just on accuracy, but on the timing of generalization relative to human developmental milestones.

Conclusion

The paper concludes that while LMs can eventually learn shared representations for filler-gap dependencies, they do so inefficiently and late in training compared to humans. Under developmental constraints, LMs rely on item-sensitive, construction-specific mechanisms, suggesting that human language acquisition likely relies on a combination of statistical learning and specific inductive biases that current LMs lack.