Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways

1. Problem Statement

The paper addresses the challenge of training performant Language Models (LMs) on human-scale data (approximately 10 million words), as opposed to the massive datasets (trillions of tokens) used by state-of-the-art Large Language Models (LLMs).

• The Core Issue: Current LLMs may rely on statistical memorization and heuristics due to their massive training data, rather than developing genuine cognitive linguistic competence. It remains unclear if models can achieve human-like linguistic understanding with data amounts comparable to what a child is exposed to.
• The Goal: The authors participated in the BabyLM Challenge, aiming to train a model that performs well on linguistic benchmarks (BLiMP, SuperGLUE, MSGS) using strictly limited data (10M words for the "small" track).
• Hypothesis: Typical LLM training regimes (long sequences, random masking, text-only) may be suboptimal for learning from small datasets. The authors propose mimicking human developmental stages to improve learning efficiency.

2. Methodology

The authors developed Lil-Bevo, a small DeBERTa-based model trained from scratch. Instead of focusing on architecture changes, they modified the training regime based on three human-inspired strategies:

A. Architecture and Tokenization

• Model: DeBERTa (encoder-based), chosen for its ability to capture syntactic/semantic features with modest data.
• Tokenizer: A custom unigram SentencePiece tokenizer trained on the BabyLM dataset combined with the MAESTRO music dataset.
  • Lil-Bevo: 16,640 vocabulary size.
  • Lil-Bevo-X (larger variant): 33,280 vocabulary size.

B. Three Core Training Strategies

1. Pretraining on Music (Non-Linguistic Data):

   • Motivation: Infants are exposed to multimodal inputs (music, visual, physical) before mastering language. Structured non-linguistic data might induce hierarchical biases useful for language.
   • Implementation: Initial pretraining (5 epochs) on a mix of text (BabyLM) and music (MAESTRO dataset, 200 hours of MIDI piano recordings converted to text tokens). Sequence length: 64 tokens.

2. Curriculum Learning (Short to Long Sequences):

   • Motivation: Caregivers speak to infants using shorter, simpler utterances that increase in complexity over time.
   • Implementation: A multi-stage training regime:
     • Stage 1: 5 epochs on Music+Text (64 tokens).
     • Stage 2: 50 epochs on Text only (128 tokens).
     • Stage 3: 2 epochs on Text only (512 tokens).
   • Note: They found that training only on short sequences (128 tokens) often outperformed training on long sequences (512 tokens) for this dataset size, avoiding overfitting.

3. Targeted Masked Language Modeling (MLM):

   • Motivation: Parents often "recast" or highlight specific grammatical errors. The authors hypothesized that focusing the model's attention on specific syntactic phenomena would accelerate learning.
   • Implementation: Instead of random masking, specific tokens essential to BLiMP subtasks were masked with higher frequency.
     • Examples: Masking "that/what" for filler-gap dependencies; "not/of-ten/probably" for Negative Polarity Items (NPI); reflexive pronouns for anaphor agreement.
     • Total masking ratio remained 15% per sample (mix of targeted and random masks).

C. Experimental Setup

• Hardware: 4 Nvidia A40 GPUs.
• Baselines: Compared against a standard DeBERTa trained on long sequences (512 tokens) and various ablation studies (e.g., "Short-only," "Music+Short," "Long-only").
• Evaluation: BLiMP (grammatical acceptability), SuperGLUE (language understanding), and MSGS.

3. Key Contributions

1. Lil-Bevo Model: A baseline model for the BabyLM challenge that integrates music pretraining, curriculum learning, and targeted masking.
2. Empirical Analysis of Training Regimes: The paper provides a systematic ablation study showing that:
   • Sequence Length: Training on shorter sequences (128 tokens) significantly outperforms training on longer sequences (512 tokens) when data is limited.
   • Music Pretraining: Offers marginal, inconsistent gains; the effect is small and may be spurious.
   • Targeted MLM: Does not improve general performance but significantly boosts performance on specific syntactic tasks where the masks were targeted (e.g., NPI Licensing).
3. Open Source: All code, models, and tokenizers are released publicly.

4. Results

• Overall Performance: The models performed above chance but significantly below larger LLMs trained on massive data.
• BLiMP Results:
  • NPI Licensing: Targeted MLM showed a massive improvement (+14.8 points over the non-targeted long-sequence baseline).
  • Argument Structure: Also saw significant gains with targeted masking.
  • General Trend: The "Short-only" model (128 tokens) generally outperformed the "Long-only" model (512 tokens) by an average of 1.8 points across tasks.
• Music Pretraining: The "Music+Short" model outperformed "Short-only" on 8/12 BLiMP subtasks but suffered a large drop (-9.1 points) on the "QA Congruence tricky" task. Statistical analysis suggested the benefit of music was small and inconsistent.
• Dynabench Score: Lil-Bevo achieved a score of 0.64, comparable to the "Music-short-long" baseline but lower than the "Music-short" variant (0.69), indicating that the final long-sequence stage did not add value in this specific configuration.

5. Significance and Future Directions

• Cognitive Modeling: The study reinforces the idea that how data is presented (curriculum, structure, focus) matters as much as the amount of data for small models. It suggests that mimicking human developmental constraints (short sequences, specific feedback) is a viable path for small-scale LMs.
• Limitations: The gains were modest. The authors acknowledge that their manipulations were "proof-of-concept" and not exhaustively optimized.
• Future Work:
  • Investigating "language-like" music (more structured) vs. raw MIDI.
  • Improving data preprocessing to respect coherent text chunks (dialogues/documents).
  • Exploring contrastive objectives to explicitly penalize ungrammatical sentences.
  • Aggregating results from multiple teams to better understand the limits of human-scale training.

Conclusion: While Lil-Bevo did not match the performance of massive LLMs, it demonstrated that training on shorter sequences is crucial for small datasets and that targeted masking can effectively teach specific syntactic rules, offering a roadmap for more efficient, human-like language model training.