💬 NLP

LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering

LangFIR is a novel method that identifies sparse, language-specific features in multilingual large language models using only monolingual data and random-token filtering, enabling highly effective language steering that outperforms existing approaches relying on expensive parallel data.

Original authors: Sing Hieng Wong, Hassan Sajjad, A. B. Siddique

Published 2026-08-04

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Sing Hieng Wong, Hassan Sajjad, A. B. Siddique

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a super-smart robot that has read almost every book in the library, but it speaks every language at once. Sometimes, you ask it to tell a story in French, and it accidentally starts speaking in Spanish or even a mix of both. This is a bit like trying to steer a massive ship; the robot is so powerful that it wants to go in many directions at once. Scientists are trying to figure out how to gently nudge the robot's brain so it stays on course and speaks only the language you want. They've discovered that the robot's "brain" isn't just one big blob of thinking; it's made of tiny, specific switches (called neurons) and hidden pathways. By finding the exact switches that control "French-ness" or "Spanish-ness," they can flip them to force the robot to speak the right language. But there's a catch: usually, to find these switches, you need a massive library of books in many languages side-by-side, which is hard to get.

This paper introduces a clever new trick called LangFIR (Language Feature Identification via Random-token Filtering). The researchers realized that to find the "French switch," you don't actually need a library of French books compared to English ones. Instead, you can use a very strange ingredient: gibberish. They took a few sentences in the target language (like French) and mixed them with sequences of random, nonsensical symbols (like "ß!4shan ?áTq7"). When they fed these into the robot, they noticed something interesting: the robot's brain lit up with many switches for both the French sentences and the gibberish. These were the "boring" switches that just react to punctuation or random shapes, not the actual language. By using the gibberish as a filter to subtract out these boring switches, they were left with a tiny, super-specific set of switches that only lit up for French. They found that these switches are incredibly sparse (there are very few of them) and are mostly located in the later layers of the robot's brain. When they used these specific switches to steer the robot, it worked better than methods that used huge parallel libraries, proving that you can control a multilingual robot using just a tiny bit of single-language data and a little bit of nonsense.

The Story of the Gibberish Filter

Think of a Large Language Model (LLM) as a giant, multi-lingual orchestra. When you ask it to play a song in Italian, it's supposed to pick up the Italian violin and start playing. But sometimes, it gets confused and starts playing the German tuba or the French flute instead. For a long time, scientists thought that to teach the orchestra to play only Italian, they needed a massive sheet of music that showed every Italian note right next to its German or French counterpart. This is called "parallel data," and it's expensive and hard to collect for every language in the world.

The authors of this paper, Wong, Sajjad, and Siddique, asked a different question: What if we don't need the parallel music? What if we can find the Italian violin just by listening to the orchestra play Italian and then listening to it play... static noise?

The Problem: Too Many Noisy Switches

Inside the robot's brain, there are millions of tiny "features" or switches that activate when it processes text. When the robot reads a sentence in Portuguese, hundreds of these switches light up. But here's the problem: many of those switches aren't actually thinking about "Portuguese." They are just reacting to things that happen in every language, like periods, commas, spaces, or the general shape of a sentence. It's like if you tried to find the "Italian" player in the orchestra, but you also picked up the person who plays whenever anyone claps their hands. You'd end up with a huge, messy list of players, and you wouldn't know who actually plays the Italian music.

Previous methods tried to solve this by comparing Portuguese to English side-by-side to see what was different. But the authors noticed that you can find the "noise" players (the ones who clap for everyone) much easier. If you play a sequence of random, nonsensical tokens (gibberish) that looks like text but has no meaning, the "clappers" will still light up. The "Italian players" won't, because gibberish isn't Italian.

The Solution: LangFIR

The team built a method called LangFIR to do exactly this. Here is how they did it, step-by-step:

The Setup: They took a small number of real sentences in a target language (say, 100 sentences in Polish).
The Gibberish: For every Polish sentence, they generated a matching sequence of random tokens from the robot's vocabulary (like "X7#9@!"). These have no meaning, but they look like text.
The Scan: They fed both the Polish sentences and the gibberish into the robot and watched which switches in the brain lit up.
The Filter: They found the switches that lit up for both the Polish and the gibberish. These were the "language-agnostic" switches (the clappers). They threw those away.
The Result: What was left? A tiny, super-sparse set of switches that only lit up for Polish.

The results were surprisingly clean. For most languages, they found fewer than 5 specific switches out of thousands that were truly responsible for the language identity. It's like finding that the entire "Polish" section of the orchestra is controlled by just four specific musicians.

Why It Matters: The Power of "Just Enough"

The most exciting part of this discovery is how little data it needs. The researchers tested this on three different robot brains (Gemma 3 1B, Gemma 3 4B, and Llama 3.1 8B) and twelve different languages. They found that they only needed about 100 sentences of the target language to find these switches. Even better, the method worked just as well with only 10 sentences.

When they used these tiny sets of switches to steer the robot, the results were impressive. In a test where the robot had to translate English into a target language without being told which one, LangFIR achieved the best average score (a combination of accuracy and translation quality) across all models.

It beat the strongest previous method that used only single-language data by up to 4.7 times.
It even beat methods that used massive parallel libraries (comparing English to the target language side-by-side) by a significant margin.

The "Ablation" Test: Proving It Works

To make sure these switches were actually doing the work and not just being lucky, the researchers did a "directional ablation." This is a fancy way of saying they took the "Polish" switches and turned them off (zeroed them out) while the robot was trying to speak Polish.

The Result: When they turned off these specific switches, the robot's ability to speak Polish crashed, but its ability to speak other languages (like German or French) stayed exactly the same.
The Location: They also found that these switches are mostly located in the later layers of the robot's brain. This confirms earlier ideas that the robot processes general concepts first and then decides on the specific language near the end.

The One Weird Exception: English

There was one language that didn't quite fit the pattern: English. When they tried to find the "English" switches, they found very few, and turning them off didn't change the robot's behavior much. The authors suggest this is because English is the "default" language for many of these models. The robot's brain is so tangled with English that general concepts are mixed in with English words, making it hard to isolate a pure "English" switch. It's like trying to find the "water" switch in a cup that is already 99% water; there's no distinct switch to pull.

What This Means for the Future

The paper suggests that language identity in these massive robots is not a giant, messy cloud. Instead, it is localized in a very small number of sparse, interpretable directions. This means we don't need to collect millions of parallel sentences to control a robot's language. We can just use a little bit of the target language and a little bit of random noise to find the exact path to steer it.

The authors are careful to note that their findings are based on specific models (under 10 billion parameters) and mostly mid-to-high resource languages. They don't claim this works for every single language in the world or for the largest models yet, but the method is a strong suggestion that we can control these complex systems with much less data than we thought.

In the end, LangFIR shows that sometimes, to find the signal, you don't need more noise; you just need the right kind of noise to filter it out. By using gibberish to clear the path, they found the hidden keys to the robot's language, proving that even in a giant, multilingual brain, the switch to "French" or "Polish" is just a tiny, lonely button waiting to be pressed.

Technical Summary: LangFIR

Problem Statement

While Large Language Models (LLMs) exhibit strong multilingual capabilities, reliably controlling the specific language of their output (multilingual generation control) remains a significant challenge. Existing representation-level steering methods, which intervene on model activations at inference time, often rely on identifying language-specific directions within the residual stream. However, current approaches typically require expensive multilingual or parallel corpora to isolate these directions. Furthermore, methods based on Sparse Autoencoders (SAEs), which decompose activations into interpretable features, still face data constraints, often needing parallel data to distinguish language-specific features from general linguistic patterns.

Methodology: LangFIR

The authors introduce LangFIR (Language Feature Identification via Random-token Filtering), a method designed to discover sparse, language-specific SAE features using only a small amount of monolingual data. The core insight is that many features activated by target-language text are not language-specific but rather respond to language-agnostic properties (e.g., punctuation, whitespace, or non-linguistic symbol patterns).

LangFIR operates in three stages:

Data Generation: For a target language, the method collects $N$ monolingual sentences. Crucially, it also generates a set of random-token sequences by sampling uniformly from the tokenizer's vocabulary (excluding special tokens). These sequences lack coherent linguistic content but share the same token-level statistical properties (e.g., punctuation patterns) as natural text.
SAE Encoding: Both the target-language sentences and the random-token sequences are passed through the LLM. Residual-stream activations at a specific layer $l$ are extracted and encoded using a pre-trained SAE to obtain latent activation vectors.
Random-Token Filtering: The method identifies features that activate frequently (above a threshold $\tau$ ) for the target-language sentences ( $S_{lang}$ ). It simultaneously identifies features that activate frequently for the random-token sequences ( $S_{rand}$ ). The language-specific feature set ( $S_{spec}$ ) is obtained by removing the language-agnostic features from the language-consistent set: $S_{spec} = S_{lang} \setminus S_{rand}$ .

A steering vector is then constructed by decoding the mean activation of the top- $k$ features in $S_{spec}$ back into the residual space and adding it to the model's activations during inference.

Key Contributions

Monolingual-Only Feature Discovery: LangFIR eliminates the need for parallel or multilingual data, requiring only a small set of target-language sentences and random-token sequences to identify language-specific SAE features.
Characterization of Language-Specific Features: The authors demonstrate that the identified features are:
- Extremely Sparse: Typically fewer than five features per language remain after filtering.
- Causally Important: Directional ablation (zeroing out these features) substantially increases cross-entropy loss specifically for the target language, with negligible effects on other languages.
- Layer-Localized: These features are most dominant and active in the later layers of the model, consistent with findings that language-specific representations emerge late in the processing hierarchy.
Superior Steering Performance: Steering vectors constructed from LangFIR features achieve the best average accuracy $\times$ BLEU scores across three models (Gemma 3 1B, Gemma 3 4B, Llama 3.1 8B), three datasets, and twelve target languages. The method outperforms the strongest monolingual baseline by up to 4.7 $\times$ and surpasses methods utilizing parallel data.

Results

Efficiency: The method is highly data-efficient, achieving stable performance with as few as 10–100 monolingual sentences.
Robustness: The feature identification process is stable across different random seeds and sample sizes. The random-token filtering step is critical; removing it degrades performance dramatically (e.g., a 19 $\times$ drop in ACC $\times$ BLEU).
Generalization: While optimized for sentence translation, LangFIR also outperforms baselines on a cross-lingual answer generation task (XQuAD), where it successfully overcomes the model's tendency to generate in English despite large English contexts.
Comparison: LangFIR consistently outperforms strong baselines, including DiffMean (which uses parallel data) and neuron-based methods, achieving the highest composite scores in the majority of experimental settings.

Significance

The paper claims that LangFIR demonstrates that language identity in multilingual LLMs is localized in a sparse set of feature directions that can be discovered without parallel data. By using random-token sequences as a proxy to filter out language-agnostic features, the method provides a clean, interpretable basis for language steering. This finding suggests that inference-time language control can be achieved in domains where multilingual resources are unavailable or costly to curate. The work also highlights the utility of SAEs in isolating causal, interpretable features for model intervention, offering a pathway to more efficient and accessible multilingual control mechanisms.