Language Steering for Multilingual In-Context Learning

1. Problem Statement

Large Language Models (LLMs) exhibit a significant performance gap between high-resource languages (like English) and low-resource languages, particularly in Multilingual In-Context Learning (ICL). In a typical cross-lingual ICL setup, a model is provided with few-shot demonstrations in English (source language) but must answer a query in a target language. Despite the model's internal knowledge of the target language, performance often degrades because the model fails to effectively "switch" its internal reasoning mode from English to the target language.

The authors hypothesize that this gap is not due to a lack of internal knowledge, but rather that language identity is encoded as a separable, structured direction in the model's activation space. If true, shifting the model's internal representations toward the target language via a simple activation offset should bridge the performance gap without retraining or additional data.

2. Methodology: Language Vectors

The authors propose Language Vectors, a training-free, inference-time steering method. The approach consists of three main steps:

A. Vector Computation (Offline)

1. Data Preparation: The authors construct a compute set ($D_{compute}$) of $N$ parallel samples. Each sample consists of $k$ question-answer pairs (few-shot format) in both the source language ($L_s$, English) and the target language ($L_t$).
2. Activation Extraction: The model processes these parallel samples. For a specific layer $t$, the hidden states for all tokens in a sample are mean-pooled to obtain a single vector representation ($h^s_i(t)$ for source, $h^t_i(t)$ for target).
3. Vector Calculation: The language steering vector $v(t)$ is computed as the mean difference between the target and source representations across all $N$ samples:
   $$v(t) = \frac{1}{N} \sum_{i=1}^{N} (h^t_i(t) - h^s_i(t))$$
   This vector represents the "direction" in the activation space that shifts the model from English processing patterns to target language patterns.

B. Inference-Time Steering

During inference, the model receives English few-shot demonstrations followed by a target language query.

1. Modification: At a specific layer $t$ and token positions $P$ (e.g., on the few-shot tokens, the question, or the entire prompt), the hidden state $h_p(t)$ is modified:
   $$h'_p(t) = h_p(t) + \alpha \cdot v(t)$$
   where $\alpha$ is a scaling hyperparameter controlling the strength of the steering.
2. Implementation: This is achieved via forward hooks, requiring no parameter updates or gradient computation.

3. Key Contributions

1. Novel Steering Approach: Introduced "Language Vectors" specifically for multilingual ICL, extending prior activation steering work (which focused on output fidelity or zero-shot tasks) to improve downstream task accuracy in few-shot settings.
2. Comprehensive Evaluation: Evaluated across three datasets (MGSM for math reasoning, MSVAMP for arithmetic, XNLI for natural language inference), 19 languages, and three models (Llama-3.1-8B, Qwen2.5-7B, Qwen2.5-14B).
3. Linguistic Structure Discovery: Demonstrated that the computed vectors encode meaningful linguistic geometry. Closely related languages (e.g., Catalan-Galician, Hindi-Urdu) form tight clusters in vector space, and vector norms correlate with typological distance from English.
4. Cross-Task Transfer: Showed that language vectors computed on one task (e.g., Math) can effectively steer performance on another (e.g., NLI), suggesting the vectors capture general language identity rather than task-specific features.

4. Experimental Results

The method was compared against two baselines:

• Source Baseline (B): English few-shot prompts (standard cross-lingual transfer).
• Multilingual Few-Shot (MFS): Few-shot examples drawn from multiple languages (diversified prompting).
• Oracle: Target language few-shot prompts (upper bound).

Key Findings:

• Consistent Improvement: The proposed method ("Ours") consistently outperformed the Source Baseline across all tasks and languages.
  • MGSM (Math Reasoning): Showed the most significant gains (e.g., +2.9% to +5.4% over baseline), indicating that structured reasoning benefits heavily from language steering.
  • XNLI & MSVAMP: Showed moderate but consistent improvements, though the MFS baseline remained competitive in some arithmetic scenarios.
• Model Agnostic: Improvements were observed across Llama-3.1 and Qwen2.5 families of varying sizes (7B to 14B).
• Layer Sensitivity: Steering was most effective in early to middle layers (e.g., layers 5–10 for reasoning, layer 10 for NLI), suggesting these layers encode the core language identity shifts.
• Cross-Task Transfer: Vectors computed on MGSM successfully improved performance on XNLI and MSVAMP (with the exception of MSVAMP $\to$ XNLI, which failed, suggesting MSVAMP vectors may encode overly task-specific mathematical patterns).

5. Significance and Implications

• Efficiency: The method requires zero training, no fine-tuning, and no additional parallel data beyond a small compute set for vector calculation. It is highly practical for low-resource settings.
• Interpretability: The results provide strong evidence that LLMs operate in a "universal semantic space" where language identity is a linear, separable direction. This supports the "Linear Representation Hypothesis" in mechanistic interpretability.
• Equity: By enabling models to leverage high-quality English demonstrations for low-resource languages without performance degradation, this approach offers a pathway to more equitable access to language technologies.
• Limitations: The method requires a held-out validation set to tune hyperparameters ($\alpha$, layer $t$, position $P$) per language, and it currently relies on the existence of parallel data for the steering vector computation.

In conclusion, the paper demonstrates that a simple activation offset can effectively redirect a model's internal "language mode," significantly improving cross-lingual reasoning and inference capabilities without modifying the model's weights.