💻 computer science

Layer-Targeted Multilingual Knowledge Erasure in Large Language Models

This paper identifies intervention depth as the critical factor behind the failure of multilingual knowledge unlearning in Large Language Models and proposes MUTE, a framework that targets intermediate, language-agnostic layers to achieve robust cross-lingual erasure while preserving model utility.

Original authors: Taoran Li, Varun Chandrasekaran, Zhiyuan Yu

Published 2026-02-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Taoran Li, Varun Chandrasekaran, Zhiyuan Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, multilingual library (the AI model) that knows everything about the world, from how to bake a cake to how to build a bomb. Sometimes, we need to remove specific dangerous books from this library so no one can read them. This process is called "unlearning."

However, there's a big problem. If you try to remove a dangerous book written in English, the AI often still knows how to build that bomb if you ask in Spanish, French, or Hindi. It's like tearing out the English pages of a manual but leaving the diagrams and instructions intact in the other language versions.

This paper, titled "Layer-Targeted Multilingual Knowledge Erasure," solves this puzzle by figuring out where inside the AI's brain to perform the "surgery."

The Problem: The Wrong Depth

The researchers discovered that AI models are built like a multi-story building, and where you intervene matters immensely:

The Basement (Shallow Layers): If you try to erase knowledge here, you knock out the foundation. The AI forgets the dangerous info, but it also forgets how to speak any language properly. It's like trying to remove a specific book by burning down the whole library. The dangerous info is gone, but so is everything else.
The Penthouse (Deep Layers): If you try to erase knowledge at the very top, you're too late. By the time the information reaches the top floor, the AI has already translated it into specific language instructions (like "write the word 'bomb' in Spanish"). If you delete the instruction here, the AI just finds another way to say it. The dangerous knowledge is still hidden in the lower floors, waiting to be used.

The Result: You either break the AI's ability to speak, or you fail to remove the dangerous knowledge.

The Solution: The "Language-Agnostic" Hallway

The researchers found a "sweet spot" in the middle of the building. They call this the Language-Agnostic Region.

Think of the AI's processing like a translation pipeline:

Bottom: "I see a word in English."
Middle: "I understand the concept of a bomb." (This is the same concept whether you say "bomba" or "bombe").
Top: "I will now write the word 'bombe' in French."

The dangerous knowledge exists as a pure concept in the middle. If you erase the concept here, it disappears for every language simultaneously because all languages converge on this same idea before splitting off into their specific words.

How They Did It (The MUTE Framework)

The team created a tool called MUTE (Multilingual Unlearning via Targeted Erasure). Here is how it works, using a simple analogy:

The Map (CKA & LRDS): Imagine the AI is a giant city. MUTE uses a special map (mathematical tools called CKA and LRDS) to find the exact "district" where all languages meet. It looks for the place where the city's layout is identical for English speakers, Spanish speakers, and Hindi speakers.
The Surgery: Once they find this specific district (a specific layer in the AI), they perform the unlearning only there. They don't touch the basement or the penthouse.
The Result: Because they removed the core concept in the shared district, the dangerous knowledge vanishes from the entire building. The AI can no longer answer questions about the bomb in any language, but it can still write poetry, solve math problems, and chat happily in all those languages.

Why This Matters

Before this, people thought you had to teach the AI to forget in every single language (English, Spanish, French, etc.), which is slow and expensive.

The Big Takeaway: You don't need to teach the AI to forget in 100 languages. You just need to find the one "shared hallway" where all 100 languages meet, remove the knowledge there, and it disappears everywhere at once.

Summary Analogy

Imagine a factory that makes toys.

Old Way: You try to stop the factory from making a specific dangerous toy by yelling at the workers at the very end of the line (Deep Layers). They just ignore you and make it anyway. Or, you try to stop it by shutting down the power to the whole factory (Shallow Layers). Now, no toys are made, including the safe ones.
MUTE Way: You find the specific machine in the middle of the factory where the raw plastic is molded into the toy's shape. You smash that specific machine. Now, the dangerous toy shape can never be formed, regardless of whether the final paint job is red, blue, or green. The factory keeps running perfectly for all the other safe toys.

This paper proves that by targeting the right "layer" in the AI's brain, we can safely remove harmful knowledge across all languages without breaking the AI's ability to communicate.

1. Problem Statement

The paper addresses a critical failure in Machine Unlearning for Large Language Models (LLMs): the inability of current methods to generalize knowledge erasure across languages.

The Issue: When hazardous knowledge (e.g., instructions for synthesizing chemicals) is erased from an LLM in one language (e.g., English), the knowledge often remains accessible when the model is queried in other languages (e.g., German, Hindi).
Current Limitations: Existing unlearning methods typically operate in a monolingual setting or apply updates globally across all layers. They fail to account for the internal multilingual structure of LLMs.
The Gap: Naive approaches either fail to erase knowledge entirely or, if they succeed in erasure, cause catastrophic collapse of the model's general capabilities in held-out languages. The paper hypothesizes that the depth (layer) at which unlearning interventions occur is the determining factor for multilingual generalization.

2. Methodology: The MUTE Framework

The authors propose MUTE (Multilingual Unlearning via Targeted Erasure), a framework that identifies and targets specific "language-agnostic" layers within the model architecture to achieve robust multilingual erasure.

A. Core Hypothesis: The Layered Structure of LLMs

Through systematic experiments, the authors identify two distinct failure modes based on intervention depth:

Shallow Layers (e.g., Layer 2): Interventions here successfully erase knowledge but collapse multilingual capabilities. Shallow layers encode fundamental linguistic primitives shared across all languages; disrupting them destroys the model's ability to process non-source languages.
Deep Layers (e.g., Layer 30): Interventions here preserve utility but fail to erase knowledge. By the time data reaches deep layers, semantic representations have already diverged into language-specific subspaces. Modifying these layers only affects token generation, not the underlying stored knowledge.

B. Identifying the "Language-Agnostic Region"

MUTE uses a two-stage process to locate the optimal intervention point:

CKA-LRDS Analysis:
- Centered Kernel Alignment (CKA): Measures the similarity of hidden representations for the same semantic concept across different languages. High CKA indicates language-invariant representations.
- Linguistic Regions Development Score (LRDS): Quantifies the degree to which representations cluster by language identity versus semantic content. Low LRDS indicates language-agnostic clustering.
Target Layer Selection:
- The framework identifies a contiguous region ( $\Lambda$ ) where CKA is high and LRDS is low.
- For Parameter-based methods (e.g., RMU): The earliest layer in $\Lambda$ is selected to disrupt the shared representation before it diverges.
- For Activation-based methods (e.g., SLUG): The deepest layer in $\Lambda$ is selected to leverage more abstract, fully-formed semantic representations.

C. Layer-Targeted Unlearning

Once the target layer ( $l^*$ ) is identified, MUTE restricts all unlearning updates (gradients or weight modifications) exclusively to that layer, freezing the rest of the network. This ensures that the shared semantic concept is modified without damaging language-specific generation capabilities or fundamental linguistic processing.

3. Key Contributions

Identification of Failure Modes: The paper characterizes the "Multilingual Amnesia" phenomenon, demonstrating that intervention depth is not a free parameter but a fundamental determinant of unlearning success.
MUTE Framework: A principled method using CKA and LRDS to automatically locate language-agnostic layers, enabling unlearning on a small set of source languages (e.g., 3) to generalize to held-out languages (e.g., 9+).
Algorithm Adaptation: The authors adapted three representative unlearning paradigms (RMU, SLUG, SimNPO) to operate strictly within the identified target layers.
Mechanistic Verification: Using Logit Lens probing, the authors provide white-box evidence that MUTE achieves genuine knowledge removal from internal representations, rather than merely suppressing output tokens (the "Erasure Illusion").

4. Experimental Results

The framework was validated across three model architectures (Llama-3.1-8B, Qwen-2.5-7B, BLOOM-7b1) and three unlearning algorithms.

Performance Metrics:
- Forget Accuracy: MUTE achieved near-perfect erasure (often <2% accuracy on the "forget" set) across all languages, including those not seen during training.
- Retain Accuracy: Unlike shallow interventions (which dropped retain accuracy to ~1% in held-out languages), MUTE preserved general capabilities, maintaining retain accuracy close to the fine-tuned baseline (e.g., ~55-60% on history/law tasks).
Cross-Architecture Consistency:
- Llama-3.1: Target layer identified at Layer 9.
- Qwen-2.5: Target layer identified at Layer 19.
- BLOOM: Target layer identified at Layer 5.
- In all cases, deep-layer interventions failed to erase knowledge, and shallow layers destroyed utility.
Comparison to Baselines:
- Global Gradient Ascent: Failed to balance erasure and utility; aggressive settings caused utility collapse, while conservative settings failed to erase knowledge.
- Single-Language Unlearning: Showed asymmetric transfer; erasing in low-resource languages (e.g., Hindi) did not generalize to high-resource languages.
- MUTE: Consistently outperformed baselines, achieving effective erasure with minimal utility degradation.

5. Significance and Implications

Safety and Compliance: This work provides a scalable solution for removing hazardous knowledge from global LLMs without requiring massive, language-specific datasets for every supported language. It addresses a critical safety gap where attackers could bypass English-trained safety filters by translating queries into low-resource languages.
Paradigm Shift: The paper shifts the focus of unlearning research from "what data to optimize on" to "where in the model to intervene."
Mechanistic Insight: It offers a deeper understanding of how multilingual LLMs organize knowledge, confirming that semantic concepts converge in intermediate layers before diverging into language-specific generation paths.
Practical Application: The MUTE framework allows organizations to update models with safety constraints using only a few source languages, significantly reducing computational costs while ensuring global safety compliance.