💬 NLP

Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation

This paper demonstrates that function vectors extracted from multilingual large language models exhibit language-agnostic properties, as translation vectors derived from English-to-target tasks successfully generalize to improve performance across unseen target languages and transfer effectively to instruction-tuned variants.

Original authors: Nurkhan Laiyk, Gerard I. Gállego, Javier Ferrando, Fajri Koto

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nurkhan Laiyk, Gerard I. Gállego, Javier Ferrando, Fajri Koto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that speaks dozens of languages. You want to teach it how to translate a specific word, like "festival," from English into Italian, French, or Russian.

Usually, you'd have to teach the robot separately for each language pair. But what if you could teach it once, and that single lesson would magically help it translate into any language?

That's exactly what this paper discovered.

The "Magic Remote Control" (Function Vectors)

The researchers are studying something called Function Vectors (FVs). Think of an FV as a magic remote control for the robot's brain.

How it's made: You show the robot a few examples of translating English to French (e.g., "dog" → "chien"). The robot's brain lights up in a specific pattern. The researchers capture that pattern and turn it into a tiny digital "key" (the FV).
How it works: When you plug this key into the robot's brain later, it forces the robot to switch into "Translation Mode."

The Big Discovery: One Key Fits All Locks

The big question was: Is this "Translation Mode" specific to French, or is it a universal mode?

If the robot's brain is like a library where every language has its own separate room, then an English→French key should only open the French room. But the researchers found something surprising:

The Universal Key: They made the key using only English→French examples.
The Test: They plugged this key into the robot and asked it to translate English words into Italian, Russian, Hindi, Arabic, and Japanese.
The Result: The key worked! Even though the robot had never seen those specific keys before, the "Translation Mode" it activated helped it guess the correct words in all those different languages.

The Analogy: Imagine you teach a chef how to chop vegetables for a French salad. You then hand them a "Chopping Mode" button. When you press it, they suddenly get really good at chopping vegetables for a Thai stir-fry, a Mexican salsa, and a Japanese sushi platter, even though you only showed them French examples. The skill of chopping is universal, even if the ingredients change.

Where is the Magic Happening?

The researchers looked inside the robot's brain (its neural layers) to see where this magic happens.

The Middle Layers: They found that the "Translation Mode" lives in the middle layers of the brain. It's not at the very beginning (where it just sees the words) and not at the very end (where it speaks). It's in the middle, where the robot is actually figuring out what the words mean.
The Shared Circuit: They discovered that the robot uses the same tiny "wires" (called attention heads) to translate into French, German, and Spanish. It's like a shared highway that all the languages use to get to the "Translation Station."

Does it Break Anything Else?

To make sure this "Translation Mode" wasn't just a lucky guess, they tried removing the key.

The Test: They took the key away and asked the robot to translate.
The Result: The robot got much worse at translating, no matter which language it was trying to speak.
The Safety Check: They then asked the robot to do other things, like answering trivia questions or writing stories. Nothing changed. The robot was still just as smart at everything else. This proves the key is a specialized tool for translation, not a general "dumbness" switch.

Does it Work for Big Sentences?

The researchers tested if this worked for whole sentences, not just single words.

The Result: It worked, but not perfectly. It was like the robot could understand the idea of the sentence and get the meaning right, but sometimes the grammar or word order got a little messy.
Why? Translating a whole sentence is like directing a movie; it requires planning the whole scene. Translating a single word is just picking a prop. The "magic key" is great at picking the right prop, but directing the whole movie is a bit harder.

The Bottom Line

This paper shows that inside these giant AI brains, translation is a universal skill.

Even though the robot speaks many different languages, the part of its brain that figures out "how to translate" is the same for everyone. You don't need a different teacher for every language; you just need to find the right "switch" (the Function Vector), and it turns on the translation superpower for the whole world.

1. Problem Statement

While prior research has established that internal representations in multilingual Large Language Models (LLMs) are often language-agnostic (encoding semantic concepts independent of specific languages), it remains unclear if this property extends to Function Vectors (FVs).

Function Vectors: Compact task representations extracted from in-context learning (ICL) demonstrations that can be added to a model's residual stream to induce specific behaviors.
The Gap: It is unknown whether an FV extracted for a specific translation direction (e.g., English $\to$ French) encodes a language-specific mapping or a general, language-agnostic translation mechanism that can transfer to other target languages (e.g., English $\to$ German, English $\to$ Hindi) without retraining or re-extraction.

2. Methodology

The authors investigate this question using machine translation (MT) as a testbed across three decoder-only multilingual LLMs: Gemma-2-2B, Llama-3.2-3B, and Tiny Aya.

A. Function Vector Extraction

Prompt Construction: They create ICL prompts consisting of $N$ demonstration pairs (Source Word $\to$ Target Word) followed by a query word.
Corruption Scheme: To extract the vector, they create "corrupted" prompts by shuffling the target words in the demonstrations, breaking the source-target mapping.
Head Selection: Using activation patching, they identify attention heads where patching the mean output of clean demonstrations onto corrupted prompts maximizes the probability of the correct answer token.
Vector Formation: The FV is the sum of the mean outputs of the top-ranked attention heads. This vector is added to the residual stream during inference.

B. Experimental Setup

Extraction: FVs are extracted from a single direction (e.g., English $\to$ Spanish).
Evaluation: The extracted FV is applied to the model to translate English words into multiple unseen target languages (including French, German, Italian, Portuguese, Russian, Arabic, Chinese, Japanese, and Hindi).
Metrics:
- $\Delta$ Rank: The change in the rank of the gold translation token in the next-token distribution (positive values indicate improvement).
- Ablation: Removing the FV component from the residual stream to measure performance degradation.
- Generalization: Testing transfer from base models to instruction-tuned variants and from word-level to sentence-level translation.

3. Key Contributions

Demonstration of Language-Agnostic FVs: The study proves that translation FVs extracted from a single language pair (English $\to$ X) successfully promote correct translations across multiple other target languages, suggesting the underlying computation is shared.
Mechanistic Evidence: The authors identify that the FV effect is concentrated in a specific band of intermediate layers and relies on a set of attention heads that are shared across different translation directions.
Robustness Validation: The effect persists when transferring FVs from base models to instruction-tuned models and shows partial generalization to sentence-level translation tasks.

4. Key Results

A. Cross-Lingual Transfer (Token-Level)

Universal Improvement: Applying an FV extracted from English $\to$ Spanish consistently improved the rank of correct translation tokens for English $\to$ French, German, Italian, Russian, Arabic, Chinese, etc.
Layer Specificity: The gains were not uniform; they were concentrated in middle layers of the transformer, consistent with prior findings on translation circuits.
Head Overlap: Analysis of the top-10 attention heads selected for different translation directions (e.g., EN-ES, EN-FR, EN-DE) revealed significant overlap (6–8 shared heads out of 10), confirming a shared mechanistic basis.

B. Ablation Studies

Specific Degradation: Removing (ablating) the FV component caused a consistent drop in translation quality across all tested target languages, not just the one used for extraction.
Task Specificity: Crucially, this ablation did not degrade performance on unrelated NLP benchmarks (e.g., ARC-Easy, BoolQ, MMLU). This confirms the FV captures a translation-specific subspace rather than general model capability.

C. Transfer and Generalization

Base to Instruction-Tuned: FVs extracted from base models successfully transferred to their instruction-tuned counterparts (e.g., Llama-3.2-3B $\to$ Llama-3.2-3B-Instruct), though the effect magnitude was slightly reduced.
Word to Sentence: When applied to sentence-level translation (using FLORES-200), the FV showed partial generalization. While it improved semantic adequacy (measured by XCOMET), it was less consistent than explicit instruction prompting and struggled with surface-form alignment (measured by BLEU), likely due to the complexity of fluency and reordering in full sentences.

5. Significance

This work extends the concept of language-agnosticity from static representations and grammatical circuits to task-level function vectors.

Theoretical Implication: It suggests that multilingual LLMs possess a unified, language-agnostic "translation engine" within their residual stream that can be steered via a single vector, regardless of the target language.
Practical Implication: This opens the door for efficient, parameter-free multilingual control. A single FV could potentially steer a model to translate into dozens of languages without needing specific vectors for each language pair, reducing the overhead of managing task-specific interventions in multilingual systems.