Bridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performance

1. Problem Statement

Multilingual Large Language Models (LLMs) face significant challenges in cross-lingual tasks (e.g., translation, cross-lingual question answering) due to two primary factors:

• Data Imbalance: Pre-training corpora are dominated by high-resource languages (e.g., English), leaving low-resource languages underrepresented. This leads to poor generalization and weaker cross-lingual transfer.
• Monolingual Bias: Standard pre-training relies on Next-Token Prediction (NTP) within a single language. While this ensures fluency, it fails to explicitly align semantic spaces across different languages, resulting in suboptimal performance in tasks requiring comprehension and generation across linguistic boundaries.

Existing solutions, such as bilingual fine-tuning or contrastive learning, often require extensive parallel data, suffer from instability, or are incompatible with decoder-only architectures (which generate text autoregressively). Furthermore, methods that rely on post-hoc alignment or specific translation instructions during pre-training can lead to overfitting and a loss of generalizability.

2. Methodology

The authors propose a novel continued pre-training framework that integrates a Cross-Lingual Mapping (CL) Task alongside standard monolingual Next-Token Prediction (NTP).

A. Dual-Objective Pre-Training

The model is trained using a joint optimization of two objectives:

1. Next-Token Prediction (NTP): The model predicts the next token in a monolingual sequence. This preserves fluency and prevents catastrophic forgetting of high-resource languages.
2. Cross-Lingual Mapping (CL): Unlike traditional translation tasks that use explicit prompts (e.g., "Translate X to Y"), the CL task trains the model to generate a target language sequence ($Y$) conditioned on a source language sequence ($X$) without explicit instruction tokens.
   • Mechanism: Given a bilingual pair $(X, Y)$, the model learns to generate $Y$ based on the semantic information in $X$ and previously generated tokens in $Y$.
   • Advantage: This approach embeds cross-lingual alignment directly into the latent space of the decoder-only model, facilitating implicit transfer and robust alignment without relying on rigid instruction formats.

The total loss function is:
$$\mathcal{L}_{PT}(\theta) = \mathcal{L}_{NTP}(\theta) + \mathcal{L}_{CL}(\theta)$$

B. Language Alignment Coefficient (LAC)

To quantitatively evaluate the effectiveness of cross-lingual alignment, the authors introduce a new metric called the Language Alignment Coefficient (LAC).

• Definition: LAC is the ratio of the average cosine similarity of sentence embeddings between language pairs to the standard deviation of these similarities across selected model layers.
• Formula:
  $$\text{LAC} = \frac{\frac{1}{|L_{sub}|} \sum_{i \in L_{sub}} \cos(E_i(X), E_i(Y))}{\text{std}(\{\cos(E_i(X), E_i(Y))\}_{i \in L_{sub}})}$$
• Significance: Unlike simple average cosine similarity, LAC accounts for stability. A high LAC indicates that the model maintains consistent alignment across different layers, making it a robust metric even for low-resource scenarios with limited test data.

3. Key Contributions

1. Cross-Lingual Mapping (CL) Task: A novel pre-training objective that explicitly models linguistic correspondences in decoder-only LLMs without requiring explicit translation instructions, thereby enhancing cross-lingual alignment while preserving monolingual fluency.
2. Language Alignment Coefficient (LAC): A novel evaluation metric that quantifies both the strength and stability of cross-lingual alignment, addressing the limitations of existing metrics that are sensitive to outliers and layer variance.
3. Comprehensive Evaluation Framework: The construction of an extensive evaluation suite covering Machine Translation (MT), Cross-Lingual Summarization (CLSum), Cross-Lingual Question Answering (CLQA), and Cross-Lingual Natural Language Understanding (CLNLU) across diverse language pairs (including low-resource pairs like Czech-Ukrainian).
4. Empirical Validation: Demonstrating that explicit cross-lingual objectives in pre-training significantly outperform baselines that rely on bilingual fine-tuning or instruction-based pre-training.

4. Experimental Results

The experiments were conducted on Llama-3-8B and BLOOM-3B models, comparing the proposed Llama_CL against baselines (Standard Llama-3-8B, NTP-only, Bilingual NTP, and Instruction-based Pre-training).

Performance Gains

• Machine Translation (MT):
  • Achieved up to 11.9 BLEU score gains on WMT datasets compared to strong baselines.
  • On the Flores-EN-CS dataset, the proposed method improved BLEU by 6.0 points over the baseline.
  • Significant improvements in COMET scores (up to +10.95 points) across various language pairs.
• Cross-Lingual Question Answering (CLQA):
  • Achieved a 6.72 point increase in BERTScore-Precision on the XQuAD dataset.
  • Outperformed baselines in Exact Match (EM) and F1 scores.
• Cross-Lingual Natural Language Understanding (CLNLU):
  • Showed a >5% increase in accuracy on the Belebele dataset across multiple language pairs.
• Cross-Lingual Summarization (CLSum):
  • While other methods (like instruction-based pre-training) degraded in summarization tasks, the proposed CL method maintained or improved performance (e.g., +0.85 ROUGE-1 on CrossSum).

Ablation Studies & Analysis

• Pre-training vs. Fine-tuning: Embedding cross-lingual tasks during pre-training ($E_{cross}$) significantly outperformed applying translation instructions only during fine-tuning ($E_{post\_mt}$) or pre-training with explicit instructions ($E_{pre\_mt}$). The latter often led to overfitting and reduced generalizability.
• Monolingual Preservation: The model maintained strong monolingual perplexity scores (e.g., English perplexity dropped from 30.1 to 25.5), proving that cross-lingual alignment does not compromise fluency.
• Generalization: The model demonstrated strong zero-shot transfer to unseen language pairs (e.g., EN-UK), outperforming baselines by significant margins.

5. Significance and Conclusion

This paper establishes that explicitly integrating cross-lingual mapping objectives during the pre-training phase is a superior strategy for enhancing multilingual LLMs compared to post-hoc alignment or instruction-based fine-tuning.

• Scalability: The method is scalable and effective for both high-resource and low-resource languages.
• Robustness: The introduction of LAC provides a more reliable way to measure alignment stability, crucial for evaluating models in data-scarce environments.
• Practical Impact: The approach offers a pathway to build more inclusive LLMs that can handle complex cross-lingual reasoning and generation without sacrificing the fluency of dominant languages.

The authors conclude that while challenges remain in complex reasoning and summarization, their dual-task pre-training paradigm effectively bridges linguistic gaps, setting a new standard for multilingual model development.