Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model

1. Problem Statement

Large Language Models (LLMs) often exhibit a performance imbalance: they excel in English (and other high-resource languages) but struggle with low-resource languages. While fine-tuning on mixed multilingual datasets is a common approach, it often leads to catastrophic forgetting, where the model's English capabilities degrade significantly when learning new languages.

The core challenge addressed in this paper is: How can we improve an LLM's Multilingual Ability (MA) without deteriorating its existing English Ability (EA)?

The authors identify a specific constraint in real-world scenarios:

1. The Phase 1 English instruction dataset is often unavailable (only the fine-tuned model weights exist).
2. Storing both old and new model weights simultaneously for training is computationally prohibitive.

2. Methodology

A. Two-Phase Continual Fine-Tuning (CFT) Setup

The authors propose a sequential fine-tuning process rather than a single-phase mixture:

• Phase 1: Fine-tune a base LLM on an English-only instruction dataset (e.g., ALPACA, OPENORCA) to establish strong English Instruction Following Ability (EA).
• Phase 2: Take the Phase 1 model and fine-tune it on a Multilingual instruction dataset (e.g., MULTIALPACA, MOPENORCA) to enhance Multilingual Ability (MA).

B. Key Hypothesis: Dataset Similarity

The central hypothesis is that the similarity of instructions between Phase 1 and Phase 2 datasets dictates whether EA is preserved.

• Similar Instructions: If the multilingual dataset encodes the same instructions as the English dataset (just translated), EA is preserved or improved.
• Dissimilar Instructions: If the instruction styles or structures differ significantly, EA suffers from catastrophic forgetting.

C. Quantifying Similarity

To validate this hypothesis, the authors introduce two metrics:

1. Dataset Embedding Similarity (DES): Uses a language-agnostic embedding model (LaBSE) to compute the dot product of average embeddings between the two datasets. Higher scores indicate higher similarity.
2. Model Parameter Difference (MPD): Measures the Euclidean distance between the parameters of models fine-tuned on the respective datasets. Lower scores indicate the datasets induce similar representation shifts.

D. Mitigation Strategies

When Phase 1 and Phase 2 datasets are dissimilar (leading to EA decline), the authors explore two strategies to mitigate forgetting:

1. Distribution Replay:
   • Generative Replay (GR): Uses the Phase 1 model to generate English responses for the English counterparts of the Phase 2 multilingual prompts. This data is mixed into Phase 2 training (5% or 10%).
   • English Replay (ER): Uses existing parallel English data from the Phase 2 distribution (if available) to replay English instructions during Phase 2.
2. Layer Freezing (LF): Freezes specific layers during Phase 2 to prevent them from being overwritten.
   • Heuristics: Random freezing (LF_H1), freezing layers that changed most in Phase 1 (LF_H2), and Spectrum (freezing layers with high Signal-to-Noise Ratio based on Marchenko-Pastur distribution analysis).

3. Experimental Setup

• Models: MISTRAL-7B and LLAMA-3-8B (both base and Instruct versions).
• Datasets:
  • English: ALPACA, OPENORCA, MISTRAL-7B-INSTRUCT, LLAMA-3-8B-INSTRUCT.
  • Multilingual: MULTIALPACA (11 languages), MOPENORCA.
• Evaluation:
  • English Ability (EA): IFEval, Alpaca Eval, MMLU, HellaSwag, XLSUM_en.
  • Multilingual Ability (MA): MLQA, XQuAD, GMMLU, XLSUM (multilingual).

4. Key Results

A. CFT vs. Dataset Mixture

The Two-Phase CFT setup consistently outperforms the standard "Dataset Mixture" approach (training on English + Multilingual data simultaneously) in terms of the combined average score of EA and MA. CFT achieves better multilingual adaptation without the same degree of English degradation seen in mixture training.

B. The Role of Similarity

• Similar Datasets (e.g., ALPACA $\to$ MULTIALPACA): When instructions are semantically similar across phases, the model retains or improves its English ability while gaining multilingual skills. The hidden representations (visualized via t-SNE) remain aligned between Phase 1 and Phase 2.
• Dissimilar Datasets (e.g., Instruct $\to$ MULTIALPACA): When instruction styles differ, the model's hidden representations diverge significantly (high variance in layer activations), leading to a significant decline in English ability (e.g., an 8% drop in IFEval scores).

C. Effectiveness of Mitigation Strategies

When dissimilar datasets are used, mitigation strategies are crucial:

• English Replay (ER_10): Demonstrated the best overall performance, effectively preserving EA while boosting MA. It acts as a strong regularizer by explicitly re-introducing English patterns.
• Generative Replay (GR): Also effective, particularly GR_5 (5% replay), showing competitive results with lower computational overhead.
• Layer Freezing: Showed mixed results. While Spectrum and LF_H2 helped preserve some EA, they sometimes hindered multilingual gains compared to replay methods.
• LoRA: As a baseline parameter-efficient method, LoRA showed forgetting of English ability, confirming that simple parameter efficiency does not solve the catastrophic forgetting problem in this context.

5. Significance and Contributions

1. First Study on CFT for Multilingual Adaptation: This is one of the first works to systematically analyze the impact of sequential instruction fine-tuning on LLM multilingual capabilities.
2. Instruction Similarity Metric: The paper establishes that the semantic similarity of instructions between phases is a critical factor in preventing catastrophic forgetting, more so than just the language of the data.
3. Practical Mitigation: It provides actionable strategies (specifically Distribution Replay) for practitioners who need to adapt models to new languages without access to original training data or the ability to store multiple model checkpoints.
4. Visualization of Representation Drift: The use of t-SNE and covariance analysis on hidden activations provides a clear geometric explanation for why forgetting occurs (separation of activation clusters) and how mitigation strategies work (re-aligning clusters).

Conclusion

The paper concludes that a Two-Phase CFT approach is superior to mixture training for multilingual adaptation, provided the instruction styles are consistent. When consistency is lacking, Distribution Replay (specifically English Replay) is the most effective strategy to maintain English proficiency while acquiring multilingual skills. This offers a viable path for maintaining a single, versatile LLM for global users without the cost of maintaining separate models for different languages.