Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads

1. Problem Statement

While recent research has identified specific attention heads in Transformer models as "retrieval heads" (responsible for accessing context information), there remains a gap in understanding how multilingual Large Language Models (LLMs) manage the transition from internal reasoning (often in a latent or source language) to generating output in a specific target language. The core problem addressed is identifying the specific neural mechanisms that govern cross-lingual reasoning and the switch to target-language generation, distinct from simple context retrieval.

2. Methodology

The authors employed a mechanistic interpretability approach to dissect the internal workings of multilingual LLMs:

• Investigation of Multilingual Retrieval: The study first analyzed existing "retrieval heads" within multilingual contexts to determine if they are language-specific or shared across languages.
• Identification of Retrieval-Transition Heads (RTH): By expanding the analysis to cross-lingual settings, the authors identified a distinct subset of attention heads termed Retrieval-Transition Heads (RTH). These heads are hypothesized to be responsible for the critical transition from latent reasoning states to the generation of text in a specific target language.
• Ablation Experiments: To validate the function of these heads, the authors performed causal intervention experiments. They systematically masked (zeroed out) the activations of:
  • Standard Retrieval Heads (RH).
  • The newly identified Retrieval-Transition Heads (RTH).
• Evaluation Framework: The impact of masking was measured across two major model families (Qwen-2.5 and Llama-3.1) using four comprehensive multilingual benchmarks:
  • MMLU-ProX: A multilingual extension of MMLU-Pro for complex reasoning.
  • MGSM: A benchmark for multilingual grade-school math problems.
  • MLQA: A multilingual machine reading comprehension dataset.
  • XQuaD: A cross-lingual question answering dataset.

3. Key Contributions

• Discovery of Shared Retrieval Mechanisms: The paper establishes that in multilingual LLMs, standard retrieval heads are often shared across multiple languages, rather than being isolated to specific linguistic contexts.
• Definition of RTHs: The authors introduce the concept of Retrieval-Transition Heads (RTH), distinguishing them from standard retrieval heads. RTHs are specifically tasked with governing the transition to target-language output during cross-lingual tasks.
• Mechanistic Insight into Multilingual Reasoning: The work isolates the specific attention heads responsible for mapping latent reasoning processes to target-language generation, providing a clearer architectural map of how multilingual LLMs operate.

4. Experimental Results

The ablation studies yielded significant findings regarding the importance of RTHs:

• Superior Impact of RTH Masking: Masking the Retrieval-Transition Heads resulted in a significantly larger performance drop compared to masking standard Retrieval Heads.
• Critical Role in Chain-of-Thought (CoT): The results indicate that RTHs are more vital than standard retrieval heads for Chain-of-Thought reasoning in multilingual settings. Without functional RTHs, the model struggles to maintain reasoning coherence while switching to the target language.
• Generalizability: These findings were consistent across both the Qwen-2.5 and Llama-3.1 model families and held true across all four evaluated benchmarks.

5. Significance

This paper advances the field of mechanistic interpretability in multilingual AI by:

• Decoupling Functions: It successfully separates the mechanism of retrieving information from the mechanism of translating reasoning into a specific language, challenging the assumption that retrieval heads handle all context-dependent tasks.
• Improving Multilingual LLM Understanding: By isolating RTHs, the work provides a new lens for understanding how LLMs handle language switching and complex reasoning simultaneously.
• Future Directions: The identification of these heads opens avenues for targeted model optimization, such as improving cross-lingual reasoning capabilities or developing more efficient fine-tuning strategies that specifically enhance the transition mechanisms without retraining the entire model.