💬 NLP

Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models

This paper proposes a cost-effective framework that enhances underrepresented languages in large language models by identifying and fine-tuning sparse, language-specific subnetworks using Language Activation Probability Entropy (LAPE), achieving superior performance with minimal parameter updates while preserving general capabilities and preventing catastrophic forgetting.

Original authors: Daniil Gurgurov, Tanja Baeumel, Josef van Genabith, Simon Ostermann

Published 2026-02-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniil Gurgurov, Tanja Baeumel, Josef van Genabith, Simon Ostermann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Size-Fits-All" Model

Imagine a giant, incredibly smart library (a Large Language Model) that has read almost every book in the world. It speaks English, Spanish, and Chinese fluently because it has read millions of books in those languages. However, if you ask it about a rare language like Maltese or Welsh, it stumbles. It's like a librarian who knows the history of the whole world but has never stepped foot in a specific small village.

Usually, to fix this, you would have to hire a new librarian and retrain them from scratch on just that village's history. This is expensive, takes forever, and often makes the librarian forget everything else they knew about the rest of the world. This is called "catastrophic forgetting."

The Solution: The "Specialized Tool" Approach

The authors of this paper propose a smarter way. Instead of retraining the whole librarian, they ask: "Which specific brain cells (neurons) in the model are responsible for this specific language?"

They found that inside these giant AI models, different languages actually use different "tools" or "sub-networks." It's like a Swiss Army knife:

One blade is for cutting.
One screwdriver is for screws.
One corkscrew is for wine.

If you want to get better at opening wine, you don't need to sharpen the whole knife. You just need to tune the corkscrew.

How They Did It: The "Language Detector"

The researchers developed a special test called LAPE (Language Activation Probability Entropy). Think of this as a metal detector that scans the AI's brain to find the specific neurons that "light up" when the model reads a specific language.

Scanning: They ran the model over text in 12 different underrepresented languages (like Maltese, Welsh, and Georgian).
Identifying: The LAPE test found that only a tiny, specific group of neurons (less than 1% of the total brain) were doing the heavy lifting for each specific language.
Tuning: They took only those specific neurons and gave them a little extra training on the target language. The rest of the model (the other 99%+) was left frozen and untouched.

The Results: Better at the New Language, Unchanged at the Old Ones

The results were surprising and very efficient:

Superior Performance: This "tuning the corkscrew" method worked better than retraining the whole model, better than just retraining the main "thinking" parts, and better than other popular shortcuts used in AI today.
No Memory Loss: Because they only touched the tiny specific part of the brain, the model didn't forget how to speak English or solve math problems. It kept its general smarts intact.
Cheap and Fast: They only had to update about 0.2% to 1% of the model's total settings. This is like changing a single lightbulb in a stadium instead of rewiring the whole building.

A Secret Discovery: How the Brain Rewires

The researchers also looked how the model learned. They found that when the model learned a new language, the changes happened mostly in the final steps of the thinking process (specifically, the "down-projection" weights).

The Analogy: Imagine a factory assembly line.

The early parts of the line (the "gate" and "up" parts) are like the workers picking up the raw materials. They stayed mostly the same.
The final part of the line (the "down" part) is where the product is packaged and labeled.
The researchers found that to learn a new language, the model mostly just changed how it packaged and labeled the information at the very end, rather than changing how it picked up the raw materials. This allowed it to speak the new language without breaking the old machinery.

Why This Matters

This paper proves that we don't need massive supercomputers to teach AI new languages. By finding the specific "language neurons" and giving them a tiny, targeted workout, we can make AI speak hundreds of languages it currently ignores, without it forgetting the ones it already knows.

The authors have even released a map of these "language neurons" for over 100 languages, giving other researchers a free blueprint to do the same thing.

Technical Summary: Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models

Problem Statement

Large Language Models (LLMs) exhibit significant performance disparities between high-resource and low-resource languages. While scaling training data and model parameters improves multilingual performance, this approach is often impractical for resource-constrained scenarios involving hundreds of languages. Existing adaptation strategies face a fundamental trade-off:

Full Fine-Tuning: Often leads to catastrophic forgetting of general-purpose capabilities and requires substantial computational resources.
Parameter-Efficient Fine-Tuning (PEFT): Methods like LoRA and adapters reduce computational costs but operate without explicit consideration of how knowledge is organized within the model, potentially modifying parameters encoding different functions indiscriminately.
Neuron-Level Approaches: Previous work has shown promise in identifying language-specific structures but has largely focused on task-specific adaptation (e.g., machine translation) or interpretability rather than systematic, targeted fine-tuning for capability injection at scale.

The central research question is whether new language capabilities can be injected into LLMs by directly fine-tuning only the sparse subset of neurons that encode language-specific processing, without degrading existing general capabilities.

Methodology

The authors propose a framework for enhancing monolingual capabilities through the targeted fine-tuning of language-specific subnetworks within Feed-Forward Network (FFN) components. The approach consists of two primary stages:

1. Neuron Identification via LAPE

The method utilizes Language Activation Probability Entropy (LAPE), an information-theoretic metric, to identify neurons that exhibit strong language-specific activation patterns.

Mechanism: For each neuron $i$ in layer $\ell$ , the probability of activation under inputs from language $k$ ( $p^k_{\ell,i}$ ) is estimated over a monolingual corpus.
Entropy Calculation: The probabilities across $K$ languages are normalized to form a distribution. The LAPE score is computed as the Shannon entropy of this distribution. Low LAPE scores indicate neurons that activate selectively for a specific language (low entropy), whereas high scores indicate language-agnostic neurons.
Selection Criteria: Neurons in the lowest $\rho$ -th percentile (set to 5%) of LAPE scores are selected, subject to an activity threshold ( $\tau_{act}$ ) to filter out uniformly inactive neurons. A neuron can belong to multiple subnetworks if it exceeds the activation threshold for multiple languages, reflecting shared linguistic features.
Focus: The study focuses on FFN components, as prior research establishes them as the primary site for factual and linguistic knowledge storage in transformers.

2. Sparse Subnetwork Fine-Tuning

Once the language-specific subnetwork $S_k$ is identified for a target language $k$ , the model is fine-tuned by updating only the weights associated with these neurons.

Parameter Scope: For a transformer with GLU-based FFN blocks, the trainable parameters for a selected neuron $i$ include the corresponding rows of the gate and up-projection matrices ( $W_{gate}[i, :], W_{up}[i, :]$ ) and the column of the down-projection matrix ( $W_{down}[:, i]$ ).
Optimization: The model minimizes the language modeling loss on target-language data while keeping all other parameters frozen.
Efficiency: This approach updates only 0.2% to 1% of the total model parameters.

Key Contributions

Systematic Framework: Introduction of a framework for enhancing underrepresented languages via sparse subnetwork fine-tuning, demonstrating that language capabilities can be injected by updating less than 1% of parameters.
Empirical Validation: Extensive experiments across three models (LLAMA-3.1-8B, MISTRAL-NEMO-12B, AYA-EXPANSE-8B) and 12 mid- and low-resource languages. The method consistently outperforms full fine-tuning, FFN-only fine-tuning, LoRA, IA3, and random-subset baselines in target-language tasks while fully preserving general capabilities.
Mechanistic Insights:
- Weight Change Asymmetry: Analysis reveals that down-projection weights in later layers undergo substantially larger updates (3–5× more than gate/up projections) compared to other components, suggesting they serve as the primary site for linguistic re-wiring.
- Cross-Lingual Alignment: Sparse fine-tuning improves cross-lingual alignment (measured by cosine similarity of residual-stream activations) more effectively than dense fine-tuning, producing more language-agnostic representations in later layers.
Resource Release: Publication of language-specific neuron identifications for over 100 languages and the adaptation pipeline, providing practical resources for the community.

Experimental Results

Target-Language Performance: The sparse subnetwork approach achieved the highest average improvements on translation (FLORES-200, OPUS-100) and comprehension/classification (BELEBELE, SIB-200) benchmarks. For instance, on FLORES, targeted subnetworks achieved average improvements of up to 5 BLEU points over the base model, outperforming full fine-tuning which often degraded performance.
General Capability Preservation: Unlike full fine-tuning and FFN-only fine-tuning, which caused catastrophic forgetting (e.g., MMLU scores dropping from ~58 to ~22), the sparse method preserved general capabilities (MMLU scores remained near baseline levels, ~57.5 vs 58.1).
Comparison with PEFT: While IA3 preserved general capabilities, its gains on target-language tasks were consistently smaller than the sparse subnetwork method. LoRA and random-subset baselines showed intermediate performance but suffered from noticeable reductions in general capabilities.
Data Efficiency: The method converges faster and reaches lower validation loss than baselines. Increasing training data size (up to 200M tokens) generally improved performance, though saturation or slight declines were observed for specific languages (e.g., Lithuanian, Maltese), suggesting subnetwork capacity limits.

Significance and Claims

The paper claims that sparse, targeted fine-tuning of language-specific neurons is a viable mechanism for capability-level language adaptation. The significance of this work lies in:

Democratizing Access: Providing a cost-effective path to extend LLMs to underrepresented languages without the computational burden of full retraining or the risk of catastrophic forgetting.
Actionable Interpretability: Demonstrating that mechanistic insights (identifying specific neurons via LAPE) can directly guide model enhancement, moving beyond post-hoc analysis to actionable optimization.
Scalability: Proving that language capabilities can be injected by updating less than 1% of parameters, a result the authors note has not been demonstrated at this scale previously.

The authors conclude that this framework offers a practical foundation for future research on LLMs for underrepresented languages, with potential extensions to other modalities and the adaptive determination of subnetwork sizes based on language typology.