What Kind of Language is Easy to Language-Model Under Curriculum Learning?

1. Problem Statement

The paper addresses a central question in computational linguistics and cognitive science: To what extent do the inductive biases of Language Models (LMs) align with the typological commonality of natural languages?

Previous studies (e.g., El-Naggar et al., 2025a,b) suggested that LMs naturally prefer word-order configurations that are common in human languages (e.g., Subject-Object-Verb), implying that learning biases might shape language evolution. However, these studies typically trained models on randomly ordered data, which does not reflect human language acquisition. Humans, particularly children, often learn via Curriculum Learning (CL)—starting with simpler structures (e.g., shorter sentences) and progressing to complex ones. The authors investigate whether introducing a developmentally motivated learning scenario (CL) alters the apparent inductive biases of LMs and whether the previously observed alignment with typology holds under these conditions.

2. Methodology

2.1. Artificial Language (AL) Corpus

The study utilizes a corpus of 96 Artificial Languages generated using Generalized Categorial Grammar (GCG).

• Parameters: Each language is defined by 7 binary parameters controlling word order (e.g., Subject-Verb, Object-Verb, Head-Initial vs. Head-Final).
• Data Generation: Sentences are generated randomly and filtered by a GCG parser to ensure grammaticality.
• Dataset Split:
  • Training: 80,000 sentences with lengths uniformly distributed between 3 and 8 words.
  • Evaluation:
    • Short Test: Length 3–8 (in-distribution).
    • Medium Test: Length 9–10 (out-of-distribution generalization).
    • Long Test: Length 11–20 (heuristic-based extension).
    • Targeted Tests: Recursive and Embedded relative clauses, plus grammaticality judgments (case and verb types).

2.2. Experimental Settings

The authors compare two training regimes across three model architectures (RNN, LSTM, Transformer):

1. Original Setting (Baseline): Training data is presented in a randomized order.
2. Curriculum Learning (CL) Setting: Training data is presented based on sentence length.
   • Stages: Six stages corresponding to sentence lengths 3, 4, 5, 6, 7, and 8.
   • Progression: Models train sequentially, starting with length 3 and moving to length 8.
   • Replay Mechanism: To prevent catastrophic forgetting, 5% of data from previous stages is injected into the current stage (e.g., Stage 3 contains 5% length-3, 5% length-4, and 90% length-5).
   • Total Volume: The total number of training tokens is kept identical to the baseline (80K) to ensure a fair comparison.

2.3. Evaluation Metrics

• Perplexity (PPL): Lower PPL indicates better modeling of a specific word order.
• Typological Alignment (TA): Calculated as the Pearson correlation coefficient between the model's average PPL and the real-world frequency of that word order.
  • Negative TA: Indicates the model prefers common word orders (better alignment).
  • Positive TA: Indicates the model prefers rare word orders.
• Generalization: Performance on Medium/Long test sets and targeted syntactic structures.

3. Key Contributions

• Integration of CL into Typological Studies: This is the first study to systematically examine how a developmentally motivated learning scenario (CL) interacts with the inductive biases of LMs regarding word-order typology.
• Re-evaluation of Inductive Bias: The study challenges the robustness of previous findings that LMs inherently align with typological commonality, showing that this alignment is highly sensitive to the order of data presentation.
• Architecture-Specific Interactions: The paper demonstrates that the impact of CL is not uniform; it interacts subtly and differently with RNN, LSTM, and Transformer architectures.

4. Key Results

4.1. Impact on Word-Order Preferences

• Shift in Bias: The introduction of CL significantly alters the word-order preferences of LMs. The correlation between PPL vectors of models trained with and without CL is generally low (often near zero), indicating that the learning scenario fundamentally changes the model's inductive bias.
• Generalization vs. Overfitting:
  • Models trained with CL showed worse PPL on Short test sets (in-distribution) compared to the baseline, suggesting less overfitting to short sentences.
  • Conversely, CL models showed comparable or better PPL on Medium and Long test sets, indicating improved length generalization.

4.2. Typological Alignment (TA)

• Degraded Alignment: Contrary to the hypothesis that CL might better simulate human learning and thus improve alignment, the results show that CL generally worsens Typological Alignment.
  • In the Short and Medium test sets, TA scores became significantly less negative (or even positive) for RNNs and LSTMs, meaning the models became less aligned with real-world typological frequencies.
  • Exception: In the Long test set, TA remained significantly negative, suggesting that typological commonality still aids generalization to very long sentences regardless of the curriculum.
• Architecture Differences:
  • Transformers: Showed relatively stable but slightly degraded alignment.
  • LSTMs/RNNs: Showed drastic shifts; for instance, LSTM TA scores flipped from negative (aligned) in the baseline to positive (misaligned) in the CL setting on the Short test set.

4.3. Targeted Evaluations

• Syntactic Generalization: In tests involving recursive and embedded relative clauses, CL models generally maintained the relative ranking of word orders but exhibited higher PPLs overall.
• Grammaticality Judgments: CL consistently reduced accuracy in case-type and verb-type grammaticality judgments. The correlation between accuracy and typological commonality also weakened or became non-significant for some models.

5. Significance and Discussion

• Revisiting Learning Biases: The findings suggest that the "inductive bias" of an LM is not a fixed property of the architecture alone but is co-determined by the learning scenario. The order of data presentation is a critical factor in shaping what a model finds "easy" to learn.
• Challenging the "Bias Shapes Language" Hypothesis: If the length-based CL is a valid proxy for human acquisition, the study implies that previous non-CL studies may have overestimated the alignment between LM biases and typological patterns. This supports alternative hypotheses that language design is driven by external factors (historical accidents, geography) rather than solely by learner biases.
• Limitations of Simple CL: The authors argue that a simple length-based curriculum might be too simplified to capture the complexity of human language acquisition. The weaker alignment under CL suggests that more sophisticated CL strategies (e.g., based on construction complexity or working memory growth) are needed to accurately simulate human learning biases.
• Future Directions: The paper calls for exploring more complex curriculum strategies and re-evaluating the metrics used to measure typological alignment to better bridge the gap between language modeling and human language science.

Conclusion

The paper concludes that Curriculum Learning substantially impacts the apparent inductive bias of Language Models. While CL improves generalization to longer sentences, it disrupts the alignment between model preferences and real-world typological frequencies observed in random-order training. This highlights the necessity of aligning LM training scenarios with human developmental patterns to draw valid conclusions about the origins of linguistic universals.