Technical Summary: Cross-Lingual Steering for Figurative Language Generation

Problem Statement

Multilingual Large Language Models (LLMs) demonstrate the ability to generate figurative language (e.g., metaphors, idioms, irony) across various languages. However, it remains unclear whether the internal neural signals driving this behavior are strictly language-specific or if they contain reusable, cross-lingual components. Distinguishing between these possibilities is critical for understanding the extent of shared semantic representations in multilingual models beyond surface-level vocabulary and syntax. Specifically, the paper investigates whether steering directions derived from figurative–literal activation differences in one language can effectively induce similar figurative generation in a different target language without retraining or fine-tuning.

Methodology

The authors employ activation steering as an intervention-based probe to test the portability of figurative language representations. The experimental pipeline involves three main stages:

1. Vector Construction:

   • Contrastive Activation Addition (CAA): For a source language $g$ and a specific figurative category $c$, the authors construct a steering vector $\hat{v}^{(l)}_{g,c}$ by calculating the normalized difference between the mean hidden-state activations of figurative examples ($D^+_{g,c}$) and literal examples ($D^-_{g}$) at a specific transformer layer $l$.
   • Data: The study utilizes balanced sets of up to 500 figurative examples and 500 monolingual literal captions (e.g., COCO captions) per language-category setting.
   • Languages & Categories: Experiments cover six languages (English, Chinese, Bengali, Spanish, Italian, German) and five figurative categories (idiom, metaphor, simile, irony, sarcasm).

2. Intervention:

   • During the generation prefill phase, the steering vector is added to the residual stream at the selected layer for each prompt token position: $h^{(l)'}_t = h^{(l)}_t + \alpha \hat{v}^{(l)}_{g,c}$.
   • The intervention strength $\alpha$ is fixed at 1.0.
   • Monolingual Steering: Vectors are applied to prompts in the same language they were derived from.
   • Zero-Shot Cross-Lingual Transfer: Vectors derived from a source language (e.g., Chinese) are applied to prompts in a different target language (e.g., English) to test for transferability.

3. Geometric Interventions:

   • Language Mean Aggregation: The authors synthesize a central cross-lingual direction by averaging normalized vectors from multiple source languages (both including and excluding the target language, known as Leave-Target-Out or LTO).
   • Residual Ablation: To test if a shared geometric core drives the behavior, the component of a native target vector that aligns with the cross-lingual aggregate is mathematically removed (projected out). The performance of this residual vector is then evaluated.

4. Evaluation:

   • Target Category Rate (TCR): The proportion of generated continuations exhibiting the target figurative phenomenon, evaluated using DeepSeek-v4-flash as an LLM-as-a-judge.
   • Coherence: A 0–4 scale rating for contextual relevance and logical consistency.
   • Statistical Analysis: Paired percentage-point differences are reported with 95% bootstrap confidence intervals and adjusted McNemar $q$-values.

Key Contributions

The paper presents three primary contributions:

1. Effective Monolingual Steering: In monolingual settings, steering vectors consistently increase the Target Category Rate (TCR) compared to unsteered baselines and random-vector controls. This effect is observed in 74 out of 96 settings, with the highest robustness for structural figures of speech like metaphor and simile.
2. Zero-Shot Cross-Lingual Portability: Source-language vectors successfully steer target-language generation in 369 out of 416 zero-shot settings. Notably, in several cases, vectors derived from other languages match or even surpass the performance of vectors derived directly from the target language. German is identified as a particularly receptive target, while Bengali shows a strong source capability but weak receptivity.
3. A Shared Geometric Basis for Transfer:
   • Aggregation: Synthesized cross-lingual aggregates (specifically the Leave-Target-Out mean) systematically rival or outperform native-language steering vectors.
   • Ablation: Removing the shared cross-lingual component from native vectors (residual ablation) sharply reduces figurative generation capabilities. This indicates that transferable steering depends on a common, reusable representational component rather than language-specific artifacts.

Results

• Category Dependence: The transferability of steering signals varies significantly by category. Metaphor and simile exhibit the most robust cross-lingual transfer, with metaphor showing a mean gain of +17.2 percentage points across routes. In contrast, sarcasm and irony, which rely heavily on pragmatic context and cultural nuance, show weaker transferability and are more resistant to cross-lingual steering.
• Asymmetry: Cross-lingual transfer is asymmetric. While some languages (like German) are highly receptive to external steering signals, others (like Bengali) are poor targets despite being potent sources. This suggests that the ability to receive a figurative signal is bottlenecked by the target language's generative priors.
• Geometry vs. Native Performance: The "Leave-Target-Out" (LTO) aggregate vectors, constructed entirely without data from the target language, achieve TCRs functionally comparable to native vectors in the vast majority of scenarios (85%–98% of tested settings).
• Coherence: Utilizing shared cross-lingual geometry does not incur additional coherence penalties compared to native steering. However, ablating the shared signal causes the model to regress toward its highly coherent, unsteered default state, confirming that the loss in figurative generation is not due to incoherence but to the removal of the specific steering signal.

Significance and Claims

The paper claims to provide direct evidence that multilingual LLMs organize complex figurative intent through a reusable but target-dependent cross-lingual signal.

• Shared Representation: The findings suggest that the internal mechanisms for generating certain types of figurative language (particularly structural ones like metaphor) are not isolated to specific languages but are grounded in a shared, language-agnostic geometric core within the model's representation space.
• Behavioral Dependence: The study establishes that while the signal is reusable, the effectiveness of applying it is strictly dependent on the target language's generative capacity. The model can conceptually organize figurative intent across languages, but the successful projection of this intent is constrained by the target language's learned priors.
• Implications for Control: These results demonstrate that cross-lingual steering is a viable mechanism for controlling fine-grained behaviors in multilingual models, offering a structural foundation for alignment and control that transcends individual language boundaries.

The authors maintain a modest stance, noting that their findings reflect behavioral patterns observable under current data availability and that fully disentangling category-specific representations from construction variance remains an important direction for future work. They also highlight limitations regarding the scarcity of parallel datasets for certain categories (e.g., simile) and the challenges of evaluating pragmatic categories like sarcasm in sentence-continuation settings.