Technical Summary: Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models

1. Problem Statement

Continual Pretraining (CPT) is a practical method for extending Large Language Models (LLMs) to new languages without the prohibitive costs of pretraining from scratch. However, naïve dense CPT often leads to catastrophic forgetting, where the model loses its original general knowledge (e.g., reasoning, reading comprehension) while acquiring new language capabilities. This issue is exacerbated in multilingual settings due to the "curse of multilinguality," which forces trade-offs between language coverage and capability preservation.

While recent paradigms like X-ELM propose training independent bilingual experts to avoid cross-language interference, this approach has not been fully explored for language family experts (grouping related languages). Furthermore, even within family-expert settings, unconstrained training causes parameter drift away from the shared initialization, degrading robustness on held-out languages and related tasks. The core challenge is how to mitigate this forgetting in a distributed, family-expert CPT setting without sacrificing language acquisition or the efficiency of parallel training.

2. Methodology

2.1 Family-Expert CPT Paradigm

The authors propose organizing training around language families rather than individual languages or a single dense model.

• Setup: Five independent experts are trained in parallel, each dedicated to a specific language family: Slavic, Germanic, Indic, Austronesian, and Romance.
• Data: Training utilizes the MADLAD-400 corpus, with a fixed budget of 5 billion tokens per family (25B total).
• Initialization: All experts start from a shared base model (Gemma-3-4B).
• Goal: This approach aims to reduce intra-expert interference while allowing for flexible, distributed scaling.

2.2 Layer-Aware Parameter Alignment Strategies

To address the parameter drift that causes forgetting, the authors introduce five strategies that constrain updates or correct weights, specifically targeting the middle layers of the transformer. This design is motivated by evidence that middle layers encode language-neutral knowledge (reasoning, world knowledge), while outer layers handle language-specific processing.

The five strategies are:

1. Layer Freezing: A hard constraint during training where the middle 19 layers are frozen, and only the first 9 and last 6 layers receive gradient updates.
2. Layer-Range L2 Regularization (L2-SP): A soft constraint adding an L2 penalty to the loss function, heavily weighting the middle layers ($\lambda_{mid} = 0.05$) and lightly weighting outer layers ($\lambda = 0.001$) to anchor them to the base weights.
3. Dense-Reverted (Train-then-Revert): A dense CPT model is trained, and then the weights of the middle layers are post-hoc reverted to the base model's pre-trained weights, keeping the updated outer layers.
4. Expert-Reverted: Similar to Dense-Reverted but applied to a trained family expert.
5. Expert Soup: After training five vanilla family experts, their weights are uniformly averaged to create a single unified model ($\theta_{soup} = \frac{1}{5}\sum \theta_f$).

2.3 Baselines

The alignment strategies are compared against two unregularized baselines:

• Dense CPT: A single model trained jointly on all 32 languages without forgetting mitigation.
• Family Expert: Vanilla family experts trained without any regularization or post-hoc correction.

2.4 Evaluation

Evaluation spans 32 training languages and held-out relatives across five families, measured on four axes:

• Language Acquisition: Perplexity on held-out MADLAD text.
• Reading Comprehension: Belebele benchmark (2-shot).
• Physical Reasoning: Global-PIQA (2-shot).
• Translation: FLORES-200 (ChrF scores).

3. Key Results

3.1 Forgetting vs. Acquisition Trade-off

Parameter alignment strategies substantially reduce forgetting with minimal cost to language acquisition:

• Comprehension & Reasoning: Layer Freezing and Layer-Range L2 best preserve downstream performance. Layer Freezing improved Belebele accuracy by an average of +1.7 percentage points (pp) over the base model, whereas naïve Dense CPT caused drops of 6.6–12.3 pp.
• Translation: Dense-Reverted yielded the strongest translation gains, averaging +20.6 ChrF over the base model. It outperformed all other strategies on Slavic, Germanic, Indic, and Romance families.
• Perplexity: Layer-Range L2 and Expert Soup consistently maintained or improved held-out perplexity, whereas naïve Family Experts degraded significantly on held-out languages (e.g., Austronesian perplexity jumped from 14.04 to 28.35).

3.2 Task-Specific Performance

No single strategy dominated across all axes:

• Layer Freezing was optimal for comprehension and reasoning tasks.
• Post-hoc Reversion (Dense/Expert-Reverted) was optimal for translation quality.
• Expert Soup provided strong comprehension retention (second best after Layer Freezing) and demonstrated that uniform averaging of family experts outperforms individual family checkpoints on held-in tasks.

3.3 Causal Analysis of Layer Drift

Through targeted interpolation experiments (sweeping $\alpha$ from base to trained weights), the authors identified the causal drivers of forgetting:

• Comprehension: Degradation in Belebele accuracy is primarily driven by middle-layer drift. Restoring middle layers to base weights recovered significant accuracy, while outer layer drift had negligible impact.
• Translation: FLORES translation quality follows a different profile. Drift in the final layers was the primary driver of translation gains; restoring middle layers had little effect on ChrF scores. This suggests translation quality depends more on output behavior and generation compatibility than the middle-layer knowledge preservation required for comprehension.

4. Contributions and Significance

The paper makes three primary contributions:

1. Family-Expert CPT: It introduces a paradigm for distributed multilingual training centered on language families, generalizing the X-ELM approach to broader language groups.
2. Layer-Aware Alignment Strategies: It proposes and benchmarks five specific strategies (including hard freezing, soft regularization, and post-hoc reversion) tailored to the family-expert setting.
3. Empirical Guidelines: It provides practical deployment guidelines linking specific strategies to target tasks. The authors demonstrate that the "acquisition–forgetting frontier" is not uniform; strategy selection must be layer-aware and task-driven.

Significance:
The work challenges the notion of a single aggregate metric for evaluating CPT. It establishes that:

• Middle-layer preservation is critical for maintaining general knowledge (comprehension/reasoning).
• Outer-layer flexibility is critical for language acquisition and translation.
• "One-size-fits-all" forgetting mitigation is suboptimal; practitioners should select alignment strategies based on whether their priority is translation, comprehension, or broad language coverage.

The authors remain modest, noting that their guidelines are derived from empirical comparison on a specific model scale (Gemma-3-4B) and corpus, and that a principled method for automatically selecting strategies for arbitrary tasks remains an open question.