BabyReasoningBench: Generating Developmentally-Inspired Reasoning Tasks for Evaluating Baby Language Models

1. Problem Statement

Traditional evaluations of Large Language Models (LLMs) rely on adult-centric benchmarks that presuppose extensive world knowledge, complex instruction following, and mature pragmatic competence. These benchmarks are ill-suited for evaluating "Baby Language Models" (BabyLMs)—models trained on developmentally plausible inputs such as child-directed speech (CDS) and early-childhood narratives.

The core problem is that standard benchmarks obscure which reasoning abilities actually emerge under developmentally constrained training distributions. There is a lack of diagnostic tools to determine if BabyLMs possess specific cognitive primitives (e.g., theory of mind, causal inference) or if their performance is merely a result of memorization or surface-level pattern matching. Furthermore, many classic developmental psychology tasks are trivial for frontier models, rendering them useless for testing small, data-efficient models.

2. Methodology

A. Benchmark Construction (BABYREASONINGBENCH)

The authors introduce BABYREASONINGBENCH, a curated collection of 19 reasoning tasks grounded in classic paradigms from developmental psychology.

• Task Categories: The benchmark covers four main domains:
  1. Theory of Mind (ToM): False-belief tasks (explicit and implicit), belief attribution.
  2. Analogical & Relational Reasoning: Relational mapping, story analogies, and structural shifts.
  3. Causal Inference: "Blicket detector" inferences, causal structure learning, and intervention selection.
  4. Core Reasoning Primitives: Transitive inference, counterfactual reasoning, category-based induction, class inclusion, and conservation tasks.
• Generation Strategy: To ensure systematic evaluation and reduce reliance on specific phrasings (which could bias small models), the authors used GPT-5.2 to generate the dataset:
  1. Seed questions were created based on 19 classic task descriptions.
  2. GPT-5.2 generated 10 variants for each seed, resulting in 11 multiple-choice questions (MCQs) per task (199 total).
  3. Manual validation was performed to ensure logical consistency and remove unreasonable contexts.
  4. Token lengths of answer choices were balanced to prevent perplexity-based classification from being confounded by length bias.

B. Experimental Setup

• Models Evaluated: Two GPT-2 based BabyLM baselines:
  • BabyLM-10M: Pretrained on 10 million tokens of child-directed speech.
  • BabyLM-100M: Pretrained on 100 million tokens of child-directed speech.
• Evaluation Metric: The models were evaluated using a conditional log-likelihood approach. For each MCQ, the model calculated the probability of the tokens in the question concatenated with each answer choice. The choice with the highest summed log-probability was selected as the prediction.

3. Key Contributions

1. Developmentally Grounded Benchmark: The first benchmark specifically designed to evaluate reasoning in models trained on child-like data, bridging the gap between AI evaluation and developmental psychology.
2. Mechanism-Sensitive Evaluation: The tasks are designed to disentangle reasoning from confounds like pragmatic interpretation, memory load, and executive function, allowing researchers to test specific mechanistic hypotheses about how reasoning emerges.
3. Scalable Generation Pipeline: A method for generating robust, variant-rich datasets using LLMs to mitigate surface-form biases in small model evaluation.

4. Results

The evaluation of the two BabyLM baselines revealed non-uniform performance and dissociations across task families:

• Overall Performance: Both models performed moderately above random chance but well below ceiling.
  • 10M Model: Average accuracy ~52.15%.
  • 100M Model: Average accuracy ~53.59%.
• Scaling Effects (Selective Gains): Increasing pretraining data from 10M to 100M tokens yielded improvements in specific domains but not universally:
  • Improved: Physical cause-effect (reached 100%), simple causal analogies, blicket detector inference, and analogical problem solving. This suggests scaling helps with short-range evidence integration and relational transfer.
  • Degraded/Unchanged: The 100M model performed worse than the 10M model on Causal Structure Learning (dropping from 81.82% to 18.18%) and several counterfactual tasks. This indicates that "larger is better" does not hold for developmentally grounded reasoning; performance is highly dependent on task framing and model-specific heuristics.
• Theory of Mind & Pragmatics:
  • Both models showed competence above floor on explicit false-belief tasks (e.g., Sally-Anne test ~63.64%), contradicting the notion that these abilities are entirely absent.
  • However, performance on pragmatics-sensitive tasks (e.g., class inclusion wording, counterfactual possibilities) remained fragile and format-dependent, suggesting these abilities are not robustly generalized.

5. Significance and Implications

• Diagnostic Value: BABYREASONINGBENCH serves as a diagnostic instrument rather than a leaderboard. It reveals that BabyLMs possess a "broad but brittle" competence, capable of solving specific causal and physical reasoning tasks but struggling with tasks requiring complex pragmatic interpretation or stable belief attribution.
• Challenging Scaling Laws: The results challenge the assumption that scaling pretraining data linearly improves all reasoning capabilities. The reversal in performance on causal structure learning suggests that larger models may overfit to specific data distributions or fail to generalize heuristics learned in smaller regimes.
• Future Directions: The paper advocates for moving beyond aggregate accuracy metrics. Future work should focus on:
  • Expanding tasks to multimodal and interactive settings (matching infant behavioral experiments).
  • Systematically manipulating wording and memory load to isolate reasoning from linguistic confounds.
  • Using these benchmarks to refine training objectives for data-efficient AI.

In conclusion, the paper establishes that while BabyLMs trained on child-directed speech can exhibit emergent reasoning abilities, these abilities are highly task-specific and sensitive to the structure of the evaluation, necessitating fine-grained, developmentally informed benchmarks to truly understand the cognitive capabilities of small language models.