Technical Summary: Cross-Lingual Activation Steering for Multilingual Language Models

Problem Statement

Large Language Models (LLMs) exhibit significant performance disparities between dominant (high-resource) and non-dominant (low-resource) languages. This gap is primarily attributed to the skew of pre-training corpora toward dominant languages, leading to imbalanced representations where shared and language-specific neurons are not optimally distributed. While expanding training data or fine-tuning models are common solutions, they are often cost-prohibitive or infeasible due to data scarcity. Existing neuron-level interventions have yielded mixed results, with some approaches attempting to overwrite activations with statistical constants, effectively erasing input-dependent signals.

Methodology: Cross-Lingual Activation Steering (CLAS)

The authors propose Cross-Lingual Activation Steering (CLAS), a training-free, inference-time intervention designed to rebalance shared and language-specific representations without modifying model weights. CLAS operates through three primary stages:

1. Parallel Input Construction: The method utilizes aligned multilingual inputs (parallel sentences across languages) to analyze neuron behavior. These inputs are used solely for measuring activations and do not update the model.
2. Neuron Categorization and Bridge Layer Selection:
   • Neurons are categorized at the dataset level (using 100 parallel samples from XQuAD) into four groups: dead, language-specific, partial-shared, and all-shared. Unlike prior work that categorizes at the instance level, this approach assigns stable categories to neurons based on aggregate statistics.
   • Analysis reveals that partial-shared neurons are most prevalent in specific "bridge layers" (e.g., layers 24–29 in LLaMA, 24–25 in Qwen). These layers are selected for intervention as they offer a balance between shared structure and language-specific processing, avoiding the highly specialized final layers.
3. Activation Steering Mechanism:
   • For non-anchor languages, CLAS applies a deterministic modification to the intermediate MLP activations ($h$) in the selected bridge layers.
   • Partial-shared adjustment: Neurons encoding cross-lingual structure are boosted: $h_1 = h \odot (1 + \beta M_{shared})$.
   • Language-specific adjustment: Neurons over-specialized to a single language are suppressed: $h_2 = h_1 \odot (1 - \gamma M_{spec})$.
   • Blending: The modified activation is blended with the original signal: $h_{final} = (1 - \alpha)h + \alpha h_2$.
   • The anchor language (typically English) remains unmodified to serve as a stable reference point. The parameters $\beta$, $\gamma$, and $\alpha$ are tuned via grid search.

Key Contributions

• CLAS Framework: Introduction of a training-free mechanism that selectively modulates neuron activations to enhance cross-lingual transfer during inference.
• Functional Divergence Insight: Discovery that effective cross-lingual transfer is driven by functional divergence rather than strict geometric alignment with the anchor language. Performance gains correlate with increased separation of language clusters in the representation space.
• Empirical Validation: Demonstration of CLAS effectiveness across diverse tasks (classification and generation) and models (LLaMA 3.1 8B and Qwen 2.5 7B), achieving improvements while maintaining stability in the anchor language.

Results

The authors evaluated CLAS on the XNLI (Natural Language Inference) and XQuAD (Question Answering) benchmarks.

• Classification (XNLI): CLAS achieved average accuracy improvements of +1.93% for LLaMA and +0.45% for Qwen over base models. These gains were statistically significant ($p < 0.05$ and $p < 0.001$, respectively). Notable improvements were observed in languages like Urdu (+7.00), Chinese (+5.89), and Hindi (+5.80) for LLaMA.
• Generation (XQuAD): CLAS improved average token-level F1 scores by +0.94 (LLaMA) and +1.10 (Qwen). However, these gains were not statistically significant due to higher variance in generative tasks. The results showed high variability, with some languages experiencing substantial gains (e.g., Russian +8.94 for Qwen) while others regressed (e.g., Hindi -5.28 for Qwen).
• Representation Analysis:
  • Alignment: Contrary to the intuition that models should align closer to English, CLAS generally reduced cosine similarity to the English anchor.
  • Cluster Separation: t-SNE visualizations revealed that CLAS reshapes representations by driving functional divergence. Languages that moved from ambiguous, overlapping regions into distinct clusters showed the largest performance gains.
  • Steering Direction: The optimal steering direction ($\alpha$) varied by task and model; some settings benefited from reinforcing shared representations (positive $\alpha$), while others required amplifying language-specific signals (negative $\alpha$).

Significance and Claims

The paper claims that targeted activation steering can unlock latent multilingual capacity in existing models without the need for additional data or parameter updates. The core significance lies in the finding that functional divergence, rather than strict alignment with a dominant language, is the key to improving performance on non-dominant languages.

The authors emphasize that their approach differs from prior neuron interventions (e.g., Mondal et al., 2025) by preserving the proportionality of the model's actual activations rather than overwriting them with static constants. This allows the model to retain its input-dependent structure while rebalancing the contribution of shared versus language-specific features.

The work concludes that while CLAS offers a promising, lightweight alternative to fine-tuning, its effectiveness is task-dependent and not uniform across all languages, highlighting the need for careful, task-aware intervention strategies.