Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs

Technical Summary: The Role of Cross-Lingual Alignment in the Multilingual Performance of LLMs

Problem Statement
Large Language Models (LLMs) demonstrate impressive multilingual capabilities, successfully performing Natural Language Understanding (NLU) tasks in languages other than English despite being predominantly trained on English text. While encoder-only architectures have shown that knowledge transfers from high-resource to low-resource languages, the mechanisms driving this generalization in decoder-only LLMs remain poorly understood. Specifically, it is unclear whether an LLM's ability to align representations of non-English inputs with their parallel English counterparts causally influences performance on individual task instances. Prior work has established correlations between cross-lingual alignment and multilingual accuracy at the language level using aggregate metrics, but these studies do not explain how representation alignment affects model behavior on specific NLU instances.

Methodology
The authors propose a framework to investigate cross-lingual alignment at the instance level and establish a causal link between alignment and task success.

1. Instance-Level Alignment Metrics:
   The paper introduces three metrics to quantify alignment for discriminative NLU tasks (Multiple-Choice Question Answering):

   • DALI (Discriminative Alignment Index): A binary score indicating whether the cosine similarity of cross-lingual matched premise-ending pairs exceeds that of mismatched pairs within a specific instance.
   • DALIst: A stricter variant of DALI that requires cross-lingual matched pairs to also exceed the similarity of mismatched pairs within the same language, mitigating false positives caused by anisotropy in transformer representations.
   • MEXAT: A task-specific variant of the existing MEXA metric, computed using task premises rather than generic parallel corpora.

2. Experimental Design:

   • Datasets: The study utilizes three multilingual NLU benchmarks (Belebele, Xstorycloze, Xcopa) containing parallel premise-option pairs across 24 languages.
   • Sample Categorization: Instances are categorized into Transfer Success (TS) (correct in both English and non-English) and Transfer Failure (TF) (correct in English, incorrect in non-English).
   • Associative Analysis: The authors compare the alignment scores (DALI, DALIst, MEXAT) of TS and TF samples across transformer layers to determine if successful transfers exhibit higher alignment.
   • Activation Patching: To establish causality, the authors perform activation patching on TF instances. They replace the hidden states ($h^\lambda$) of a failing non-English forward pass with the corresponding states from a successful English forward pass at specific layers ($\lambda$).
   • Control Experiments: To ensure patches are correcting based on semantic alignment rather than simply injecting the target token (e.g., the answer choice 'A' or 'B'), the authors perform control patching using unrelated English samples that share the same target token but different semantics.

Key Results

• Association with Success: Across 33 language-benchmark-metric combinations, Transfer Success (TS) instances consistently exhibit higher cross-lingual alignment scores than Transfer Failure (TF) instances. This association is statistically significant in 8 cases and positive in 30, suggesting that better alignment correlates with successful generalization.
• Layer-Specific Effects: The analysis reveals that cross-lingual alignment peaks in the middle layers of the transformer (approximately layers 14–25).
• Causal Evidence via Patching:
  • Patching English activations into the middle layers of failing non-English instances significantly increases the probability of the model flipping to the correct answer.
  • The "flip rate" (percentage of corrected predictions) is highest when patching occurs in these middle layers.
  • Semantic Specificity: Control experiments demonstrate that semantically equivalent English patches are significantly more effective at correcting predictions than control patches (unrelated English samples with the same answer token). This confirms that the correction is driven by semantic alignment, not merely the injection of the target token.
• Entropy Analysis: Corrections achieved through semantically equivalent patches result in lower output entropy (higher confidence) compared to control patches, suggesting that alignment stabilizes the model's decision-making process.

Key Contributions

1. Instance-Level Metrics: The introduction of DALI and DALIst, which move beyond language-level aggregate analysis to quantify alignment on a per-instance basis for discriminative tasks.
2. Causal Validation: Providing causal evidence through activation patching that misalignment in middle layers is a primary driver of cross-lingual failures. The study demonstrates that "forcing" alignment by injecting English representations can rectify non-English errors.
3. Mechanism Clarification: The work supports the hypothesis that decoder-only LLMs process non-English text by implicitly pivoting through English representations in the middle layers, where language-agnostic reasoning occurs.

Significance and Claims
The paper claims to bridge the gap between correlational studies of multilingual alignment and causal mechanisms of model behavior. By demonstrating that cross-lingual semantic alignment in localized middle layers causally contributes to correctness, the authors argue that the "pivot through English" is not just a statistical artifact but a functional mechanism for reasoning.

The authors modestly frame their findings as evidence that alignment is a necessary component for successful transfer in discriminative tasks. They note that while patching can correct errors, it does not fix all instances, implying that other factors may also contribute to failure. The paper concludes that these findings motivate future work on inference-time steering to leverage high-fidelity English activations for improving non-English performance, while also cautioning about potential downsides such as inflated confidence or the erasure of local non-English knowledge. The study is limited to discriminative NLU tasks and bilingual alignment (non-English to English), excluding alignment between other language pairs.