Exemplar Retrieval Without Overhypothesis Induction: Limits of Distributional Sequence Learning in Early Word Learning

1. Problem Statement

The paper addresses a fundamental question in cognitive science and developmental psychology: How do learners transition from memorizing specific facts (first-order learning) to forming abstract rules about category structures (second-order learning/overhypotheses)?

• The Overhypothesis: Children do not just learn that "ball" is round; they learn that shape is the defining feature for object categories. This allows them to generalize to novel categories after minimal exposure.
• The Gap: While hierarchical Bayesian models (e.g., Kemp et al., 2007) can formalize this induction, it remains unclear if distributional sequence learning (the mechanism behind modern autoregressive transformers) is sufficient to produce this leap without explicit meta-learning scaffolds or multimodal grounding.
• The Hypothesis: The study tests whether autoregressive transformers trained on synthetic corpora with clear statistical regularities can spontaneously induce overhypotheses (specifically the "shape bias") or if they are limited to exemplar memorization.

2. Methodology

A. Experimental Design

The study is a pre-registered computational experiment involving 155 runs (120 primary + 30 dose-response + 5 exploratory).

• Models: Three autoregressive transformer sizes were trained from scratch:
  • Tiny: ~3.4M parameters (4 layers, 4 heads, $d_{model}=256$)
  • Small: ~10M parameters (6 layers, 8 heads, $d_{model}=512$)
  • Medium: ~25.6M parameters (8 layers, 8 heads, $d_{model}=768$)
• Corpus Construction: Synthetic corpora were generated using nonce words.
  • Structure: Sentences followed templates like "A [noun] is a [shape] [color] [texture] thing."
  • Regular Condition: Shape was the stable feature within a category (e.g., all "blickets" were "round"), while color and texture varied. This creates the statistical signal for overhypothesis induction.
  • Control Conditions: Eight conditions were designed to rule out alternative explanations, including:
    • Scrambled: Stable features randomized per category (destroying cross-category regularity).
    • Feature-swap: Different domains had different stable features (Shape vs. Texture).
    • Label manipulations: Varying the presence of noun labels (25%, paraphrased, or absent).
    • Noise injection: Random tokens inserted to simulate ambiguity.

B. Evaluation: The Wug Test Battery

A 1,040-item battery evaluated the models using forced-choice log-probability comparisons.

• First-Order (FO) Task: Predict the correct feature for a trained noun. (Tests exemplar retrieval).
• Second-Order (SO) Task: Predict the correct feature for a novel noun (never seen in training). (Tests overhypothesis induction).
• Diagnostic Tasks:
  • Frame-variant: Tests generalization to unseen syntactic structures.
  • Feature-swap: Tests if the model relies on the noun or the syntactic frame to determine the feature.
  • Linear Probes: Analyzed internal representations to see if noun-shape associations were encoded.

3. Key Results

A. Primary Finding: The Dissociation

There was a stark dissociation between first-order and second-order performance:

• First-Order (Exemplar Retrieval): 100% accuracy. All models perfectly memorized the noun-feature mappings for trained items.
• Second-Order (Overhypothesis Induction): ~50–52% accuracy (Chance). Models failed to generalize the "shape is stable" rule to novel nouns. Performance was statistically equivalent to chance across all model sizes and conditions (confirmed by TOST equivalence testing).

B. Mechanistic Diagnosis: Template Matching

The study identified why the models failed:

• Frame-Conditioning: In the "Feature-swap" condition, models achieved near-ceiling accuracy (98–100%) when the syntactic frame was present but dropped to below chance (14–24%) when only the noun was provided.
• Conclusion: Models rely on frame-to-feature template matching (e.g., "If the sentence structure is X, the answer is Y") rather than learning a structured noun → category → feature abstraction. They learn surface-level regularities to minimize loss, not deep category structures.

C. Representational Collapse

• Novel Nouns: Embeddings for novel nouns were nearly indistinguishable (cosine similarity $\approx 0.999$), whereas trained nouns had distinct representations.
• Cause: Because novel nouns were never seen during training, their embeddings were never updated. Without distinct embeddings, the model cannot differentiate between novel categories to apply a learned rule.

D. Ideal Observer Comparison

A Hierarchical Bayesian Model (HBM) trained on the same data successfully recovered the overhypothesis (low $\alpha$ parameter), proving the training data contained the necessary signal. The failure was specific to the learning algorithm (autoregressive next-token prediction), not the data.

4. Key Contributions

1. Empirical Limit of Distributional Learning: The study provides rigorous evidence that standard autoregressive pretraining, even at developmentally plausible scales, is insufficient for overhypothesis induction. It supports exemplar memorization but fails at hierarchical abstraction.
2. Methodological Innovation:
   • Development of a Wug Test Battery for computational models that explicitly separates first-order and second-order learning.
   • Introduction of the Feature-Swap Diagnostic to distinguish between genuine category inference and syntactic template shortcuts.
3. Mechanistic Insight: Demonstrated that the failure is due to shortcut learning (template matching) and representational collapse of novel tokens, rather than a lack of capacity or data diversity in the specific setup.
4. Theoretical Constraint: The results constrain statistical learning theories of word learning, suggesting that additional mechanisms (e.g., meta-learning, structural priors, or multimodal grounding) are required to explain how children form overhypotheses.

5. Significance and Implications

• For Cognitive Science: The findings challenge the notion that distributional statistics alone can bootstrap the "shape bias" or other inductive biases in early word learning. The transition from "knowing facts" to "knowing rules" likely requires computational structures beyond simple next-token prediction.
• For AI and LLMs: The study highlights a specific failure mode in small-scale language models: the tendency to learn positional/syntactic shortcuts rather than semantic generalizations when training data is highly regular. It suggests that without meta-learning or diverse cross-task exposure, models may not develop robust inductive biases.
• Future Directions: The paper identifies the need for new architectures or training regimes (e.g., meta-learning, increased input diversity, or multimodal grounding) to bridge the gap between exemplar retrieval and structured induction.

In summary, the paper concludes that autoregressive distributional sequence learning is a necessary but insufficient condition for overhypothesis induction, as it reliably produces memorization but fails to extract the higher-order structural regularities required for abstract generalization.