COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Sampling

1. Problem Statement

Large Language Models (LLMs) exhibit significant performance disparities across languages, often suffering from negative cross-lingual interference when naively fine-tuned on multilingual data.

• The Core Issue: Training data distributions often misalign with real-world usage distributions (live production interactions). When models are exposed to irrelevant patterns or under-exposed to crucial ones for a target language, they learn suboptimal representations.
• The Trade-off:
  • Monolingual Fine-tuning: Avoids interference but incurs high memory overhead (one model per language) and lacks cross-lingual transfer benefits.
  • Naive Multilingual Fine-tuning: Uses all available data but often degrades performance on specific languages due to "noise" from unrelated languages.
• The Gap: Existing methods often rely on static linguistic similarity (e.g., picking languages based on family trees) or computationally expensive gradient-based filtering. There is a lack of efficient, data-centric strategies that dynamically select cross-lingual data based on semantic distribution gaps to maximize positive transfer while minimizing interference.

2. Methodology: COMPASS

COMPASS is a data-centric framework that adapts pre-trained LLMs to target languages using Parameter-Efficient Fine-Tuning (PEFT) and Adaptive Semantic Sampling.

A. Core Architecture

• Base Model: A single shared pre-trained multilingual model ($M_{base}$).
• Adapters: Lightweight, language-specific adapters (using DoRA - Weight-Decomposed Low-Rank Adaptation) are trained for each target language. This allows for efficient storage and deployment without retraining the full model.
• Routing: An inexpensive language identification model routes incoming queries to the specific language adapter.

B. The Sampling Strategy (Initial Adaptation)

The core innovation is a distribution-aware sampling strategy that selects a subset of auxiliary multilingual data ($D_{aux}$) to augment the target language's training data ($D_t$).

1. Embedding & Clustering: All data (target, auxiliary, and a proxy for live usage $E_t$) are encoded using a multilingual embedding model (e.g., Jina-Embeddings-v3). The embeddings are clustered (e.g., using HDBScan) into semantic groups ($C_k$).
2. Distribution Mismatch Calculation: The system calculates the ratio ($\rho_k$) between the density of a cluster in the evaluation/usage set versus the target training set.
   • If $\rho_k > 1$, the cluster is under-represented in training (a "hole").
   • If $\rho_k < 1$, the cluster is over-represented.
3. Cluster-Level Weighting: Auxiliary data is sampled with weights proportional to the mismatch ratio. Clusters with high gaps receive more auxiliary samples.
4. Instance-Level Weighting (Curriculum Learning): Within a selected cluster, the sampling strategy adapts based on the cluster's coverage status:
   • High Mismatch: Prioritizes prototypical examples (close to the cluster centroid) to establish a strong semantic foundation.
   • Low Mismatch: Gradually introduces boundary examples (further from the centroid) to improve robustness and handle ambiguity.
5. Diversity Penalty: To prevent redundancy, a similarity threshold is applied; if an instance is sampled, similar instances in the same cluster are penalized to ensure semantic diversity.

C. Continual Learning: COMPASS-ECDA

To address distribution shifts in production (e.g., new jargon, seasonal trends), COMPASS extends into COMPASS-ECDA (Elastic Consolidation and Distributional Anchoring):

• Shift Detection: Monitors incoming data streams using Jensen-Shannon (JS) divergence between the current cluster distribution and the reference distribution. A significant divergence triggers an update.
• Elastic Consolidation & Distributional Anchoring (ECDA):
  • Rehearsal: Instead of random replay, the system uses "Distributional Anchors"—prototypical examples from the centroids of stable, previously learned clusters.
  • Loss Function: The update minimizes a composite loss:
    $$L_{total} = L_{task}(D_{new}) + \beta \cdot L_{DAR}(B_{anchor}) + \lambda \cdot L_{EWC}$$
    Where $L_{DAR}$ is the task loss on anchors (preventing forgetting of core concepts) and $L_{EWC}$ (Elastic Weight Consolidation) penalizes changes to parameters critical for the previous distribution.

3. Key Contributions

1. Novel Sampling Strategy: A distribution-guided, unsupervised method for cross-lingual data selection that identifies and fills semantic gaps, outperforming linguistic similarity baselines.
2. Efficient PEFT Integration: Demonstrates that lightweight DoRA adapters, when combined with strategic data selection, can match or exceed full fine-tuning performance with significantly lower computational and storage costs.
3. Continual Adaptation Framework: Introduces COMPASS-ECDA, which successfully balances plasticity (learning new distributions) and stability (preventing catastrophic forgetting) in dynamic production environments.
4. Extensive Empirical Validation: Validated across three diverse architectures (Phi-4-Mini, Llama-3.1-8B, Qwen2.5-7B) and multiple benchmarks (Global-MMLU, MMLU-ProX, OneRuler).

4. Experimental Results

• Performance Gains: COMPASS consistently outperforms baselines (Monolingual, Random Sampling, LangRank, LangSim, and "All-data" multilingual training).
  • On Global-MMLU, COMPASS improved Phi-4-Mini by +8.9% over the monolingual baseline and +13.6% over the "All-data" baseline (which suffered negative transfer).
  • It significantly outperformed linguistic similarity baselines (LangRank/LangSim) by ~3-5%, proving that distribution alignment is more critical than language family proximity.
• Low-Resource Languages: The method provided the most significant gains for low-resource languages (e.g., Swahili, Yoruba, Bengali), where targeted cross-lingual transfer is most beneficial.
• Long-Context Transfer: On the OneRuler benchmark (up to 128k context), COMPASS-enabled models showed improved long-context reasoning in low-resource languages, suggesting that better semantic alignment allows the model to leverage pre-trained long-context mechanisms more effectively.
• Continual Learning: In multi-step shift simulations, COMPASS-ECDA achieved Pareto-optimal performance, maintaining high accuracy on new tasks (MMLU-ProX) while retaining >95% performance on original tasks (Global-MMLU), significantly outperforming Naive Fine-tuning (which suffered catastrophic forgetting) and EWC alone.
• Ablation Studies:
  • Embedding Quality: The method is highly sensitive to the quality of the embedding model; poor encoders lead to "indiscriminate sampling" and performance degradation.
  • Clustering: HDBScan outperformed K-Means and Taylor-Butina due to its ability to handle variable cluster densities and noise.
  • Budget: Optimal performance was achieved with an auxiliary data budget ($B$) of roughly 80% of the target data size; excessive budgets led to diminishing returns or negative transfer.

5. Significance and Impact

• Solving Linguistic Inequality: COMPASS offers a scalable, cost-effective solution to improve LLM performance in low-resource languages without requiring massive, language-specific pre-training datasets.
• Data-Centric AI: It shifts the paradigm from "more data" to "better data," demonstrating that strategic selection based on distributional gaps is superior to brute-force multilingual training.
• Sustainable Deployment: By integrating continual learning, COMPASS addresses the reality that user data distributions are non-stationary, providing a framework for models that evolve with their users without forgetting prior knowledge.
• Limitations & Future Work: The method is constrained by the base model's tokenizer efficiency and the quality of the embedding encoder. Future work suggests integrating fairness constraints to prevent bias amplification and exploring merged adapters for composable skills.

In summary, COMPASS provides a robust, efficient, and sustainable framework for developing high-performing multilingual LLMs by treating data selection as a distributional alignment problem rather than a linguistic one.