BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop

Based on the provided Call for Papers (CfP) for the 2026 BabyLM Workshop and Challenge, here is a detailed technical summary. Note that as this is a CfP for a future event (2026), the "Results" section describes the evaluation framework and baselines rather than final experimental outcomes, as the competition has not yet concluded.

1. Problem Statement

The BabyLM initiative addresses the enduring question in cognitive science and computer science: How can a computational system learn language from limited inputs?

• Cognitive Gap: Current Large Language Models (LLMs) are trained on massive datasets (trillions of tokens), which is developmentally implausible compared to human children who acquire language from limited exposure.
• Efficiency Gap: There is a need to bridge cognitive modeling with machine learning to create sample-efficient models that mimic human learning trajectories without requiring massive compute resources.
• Linguistic Bias: Previous iterations of the BabyLM Challenge were English-only, failing to address multilingual acquisition, typological diversity, and low-resource language learning.

2. Methodology & Framework

The 2026 iteration introduces a structured competition with specific constraints and a new multilingual focus.

A. Competition Tracks

The challenge features three tracks with strict data constraints:

1. STRICT: Train models on ≤100 million words.
2. STRICT-SMALL: Train models on ≤10 million words.
3. MULTILINGUAL (New): Train models on a mixture of English, Dutch, and Chinese, totaling ≤100 million tokens.
   • Token Adjustment: Word counts are adjusted using Byte Premiums (English: 1.0, Dutch: 1.0516, Chinese: 0.9894) to account for orthographic and morphological differences in UTF-8 encoding.
   • Goal: To study cross-lingual transfer between typologically similar (English/Dutch) and distant (English/Chinese) languages.

B. Training Constraints & Rules

• Epoch Limit: Models cannot train for more than 10 epochs over their training data. This prevents overfitting and mimics the limited exposure of human learners.
• Intermediate Checkpoints: Participants must submit checkpoints at specific intervals (every 1M words up to 10M, every 10M up to 100M, etc.) to analyze learning dynamics and speed.
• Data Curation:
  • Participants can use the provided detoxified BabyLM corpus (updated to remove toxic/hate speech found in 2024/2025 datasets) or swap in custom data sources, provided the total word count adheres to track limits.
  • Multimodal & Interaction: The dedicated "Multimodal" and "Interaction" tracks have been merged into STRICT and STRICT-SMALL. Participants may now use image-text pairs or "external teacher models" (simulating caregiver interaction) within the standard tracks, provided the data/token limits are respected.
• Distillation Prohibition: Knowledge distillation (revealing weights or output distributions of an external model) is not allowed unless the external model's training data counts toward the submission model's word limit.

C. Evaluation Pipeline

• Tasks: Evaluation focuses on functional and formal linguistic competence using a mix of zero-shot prompting and fine-tuning.
• Languages (Multilingual Track): English, Dutch, and Chinese.
• Hidden Tasks: A set of hidden evaluation tasks will be released shortly before the deadline to test generalization on unseen phenomena.
• Baselines:
  • STRICT/STRICT-SMALL: GPT-BERT, SimPO-based preference optimization (from previous Interaction track), and GPT-2 Small.
  • Multilingual: Naive adaptations of GPT-BERT and GPT-2 Small trained on a subsampled, byte-premium-adjusted mixture of the three languages.

3. Key Contributions

The 2026 Call for Papers introduces several significant shifts in the BabyLM ecosystem:

1. Multilingual Expansion: The debut of the MULTILINGUAL track moves the challenge beyond English, explicitly targeting typological diversity (English, Dutch, Chinese) to investigate cross-lingual transfer and script variation.
2. Dataset Remediation: Release of a detoxified training corpus addressing the presence of hate speech and toxic content in previous datasets, ensuring safer multi-turn generation and interaction simulations.
3. Track Consolidation: Merging of Multimodal and Interaction tracks into the main tracks. This acknowledges that these capabilities can be integrated into standard small-scale training without needing separate evaluation silos, while still allowing for complex training regimes (e.g., RL with external teachers).
4. Refined Constraints: Formalization of Byte Premiums for multilingual token counting and strict enforcement of the 10-epoch limit to ensure developmental plausibility and democratize access (preventing well-funded groups from dominating via massive compute).
5. Cognitive Modeling Alignment: The requirement for intermediate checkpoints allows researchers to compare model learning curves directly with child language acquisition trajectories.

4. Results (Current Status & Expected Outcomes)

• Status: As this is a Call for Papers for the 2026 workshop (scheduled for October 2026), final competition results are not yet available.
• Expected Outcomes:
  • Identification of optimal data mixtures for multilingual small-scale training.
  • Analysis of how typological distance (e.g., English vs. Chinese) affects sample efficiency in small models.
  • Validation of whether "interaction" with external teacher models improves performance within strict data limits compared to passive training.
  • Benchmarking of new architectures against the GPT-BERT and SimPO baselines under 100M/10M word constraints.

5. Significance

The BabyLM 2026 workshop is significant for several reasons:

• Democratization of AI Research: By capping data and epochs, the challenge levels the playing field, allowing researchers with limited compute resources to contribute meaningfully to language model pretraining.
• Bridging Disciplines: It actively fosters collaboration between cognitive scientists (studying human acquisition) and NLP researchers (building efficient models), using the "small data" constraint as a shared experimental paradigm.
• Inclusivity in NLP: The shift to multilingual training addresses the historical English-centric bias in LLM research, promoting the development of technologies for underrepresented and low-resource languages.
• Safety and Ethics: The proactive detoxification of the training corpus sets a new standard for ethical data practices in small-scale model training, ensuring that safety is not an afterthought.
• Scientific Rigor: The focus on learning dynamics (via checkpoints) rather than just final performance metrics provides deeper insights into how models learn, not just what they learn.