💬 NLP

Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

This paper addresses the limitations of existing Multilingual Machine Unlearning (MMU) evaluations by proposing two new metrics, the Knowledge Separability Score (KSS) and Knowledge Persistence Score (KPS), to better assess cross-linguistic information removal and provide deeper insights into MMU phenomena.

Original authors: Kyomin Hwang, Hyeonjin Kim, Sangyeon Cho, Nojun Kwak

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kyomin Hwang, Hyeonjin Kim, Sangyeon Cho, Nojun Kwak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Echo Chamber" Effect

Imagine you have a very smart, multilingual librarian (the AI) who has read millions of books in English, Spanish, Chinese, and many other languages. One day, you realize the librarian has memorized a secret about a specific person, "Veronica Bowman," and you want that secret erased from their mind for privacy reasons.

In the past, researchers tried to make the librarian forget this secret by only speaking to them in English. They would say, "Forget everything about Veronica Bowman."

The Catch: The paper argues that this approach is like trying to clean a room by only wiping the English corner. Because the librarian learned the secret in many languages, the information might have "spread" like a rumor. Even if the librarian forgets the secret in English, they might still remember it perfectly well when you ask in Spanish or Swahili. The secret has "crossed the border" into other languages inside the librarian's brain.

The Old Way of Testing (The Flawed Mirror)

Previously, to check if the librarian had forgotten the secret, researchers would ask questions in one language at a time.

Ask in English: "Do you know Veronica?" -> Librarian: "No." (Success!)
Ask in Spanish: "Do you know Veronica?" -> Librarian: "Yes." (Failure!)

The old method would only look at the English answer and declare, "Great job, the secret is gone!" The paper says this is misleading. It's like checking if a leaky boat is dry by only looking at the front deck, while the back is still flooding.

The New Solution: Two New "Thermometers"

To fix this, the authors created a new way to test the librarian and two new "thermometers" (metrics) to measure how well the forgetting actually worked across all languages.

1. The "Knowledge Separability Score" (KSS)

The Analogy: Imagine you have a bag of red marbles (the secret you want to forget) and blue marbles (safe information you want to keep).

The Goal: You want the librarian to be able to perfectly tell the red marbles from the blue ones.
What KSS Measures: It checks if the librarian can clearly distinguish between "things to forget" and "things to keep" across all languages at once. If the librarian is confused and thinks some blue marbles are red (forgetting safe info) or some red marbles are blue (keeping the secret), the score goes down.
The Paper's Finding: They found that some methods (like "pruning," which is like cutting out parts of the librarian's brain) were good at separating the marbles in general, but they were messy and accidentally threw away some safe blue marbles too.

2. The "Knowledge Persistence Score" (KPS)

The Analogy: This is the "Rumor Test."

The Scenario: You tell the librarian to forget the secret in English.
The Test: You then ask the librarian in a language they didn't use to learn the secret (a "hold-out" language).
What KPS Measures: It measures how much the secret "persisted" or leaked into that new language. If the librarian still knows the secret in the new language, the score is high (which is bad). If they truly forgot it everywhere, the score is low (which is good).
The Paper's Finding: They discovered that even when the librarian seemed to forget the secret in the training languages, the secret often "persisted" in other languages. The secret had traveled across the language borders and was still hiding there.

How They Did the Experiment

To prove this, the authors didn't use real people's secrets (which would be unethical). Instead, they built a massive, fake library:

Created 200 Fake People: They invented 200 fictional characters with made-up names, birthdays, and hobbies.
Translated Everything: They wrote questions and answers about these fake people in 10 different languages (including big ones like English and Chinese, and smaller ones like Albanian and Bengali).
The "Forget" Game: They trained an AI on this fake data, then tried to make it forget specific people using different techniques.
The Test: They used their two new thermometers (KSS and KPS) to see if the AI truly forgot the fake people across all 10 languages.

The Main Takeaways

Language isn't isolated: Information in AI doesn't stay in one language. If you teach an AI a secret in English, it often learns it in Spanish, French, and others automatically.
Old tests are lying to us: Checking if an AI forgot something in just one language isn't enough. You have to check all languages to be sure the secret is truly gone.
The "Hold-Out" Danger: The paper found that secrets often hide in languages the AI wasn't explicitly trained to forget. Even if you scrub the memory clean in English, the "echo" of that secret can still be heard in other languages.

Conclusion

The paper concludes that we need to stop treating languages as separate rooms and start treating them as one big, connected house. To truly protect privacy in AI, we need new tools (like KSS and KPS) that check the whole house, not just one room, to make sure the secret is actually gone.

Technical Summary: Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

1. Problem Statement

Large Language Models (LLMs) trained on multilingual corpora pose privacy risks, necessitating Multilingual Machine Unlearning (MMU) to remove sensitive information across multiple languages. While existing Machine Unlearning (MU) research has focused on English-centric datasets, recent studies highlight that unlearning in multilingual settings is complicated by cross-linguistic spread, where knowledge acquired in one language persists or spreads to others.

Current evaluation protocols for MMU are insufficient because they:

Evaluate languages in isolation: They assess unlearning performance per language (e.g., checking if "Veronica Bowman" is forgotten in English) without verifying if the same knowledge remains accessible in other languages (e.g., Albanian or Afrikaans).
Fail to capture distribution: They do not account for the fact that knowledge in MMU is distributed across languages and acquired through both direct memorization (training on specific language pairs) and indirect cross-linguistic spread.
Overstate effectiveness: A model may appear to have successfully unlearned information in the evaluated language while the underlying information remains retrievable in a different language, leading to misleading conclusions about privacy safety.

2. Methodology

Dataset Construction

To address the lack of standardized evaluation data, the authors constructed a large-scale multilingual parallel dataset:

Scale: 3,800 synthetic QA pairs across 10 languages (19 attributes × 200 profiles).
Languages: Includes 5 high-resource (English, Chinese, German, Russian, Spanish) and 5 low-resource languages (Bengali, Hebrew, Tamil, Afrikaans, Albanian).
Generation Pipeline: Synthetic profiles were generated using Faker, converted to English QA pairs using an LLM, and translated into 9 other languages. A back-translation verification process (using Google Translate and LLMs) ensured semantic equivalence across all languages.
Experimental Setup: The dataset was split into Training Languages (used for memorization and unlearning) and Hold-out Languages (used exclusively for evaluation to simulate cross-linguistic spread).

Proposed Metrics

The paper introduces two novel metrics designed to evaluate the multilingual nature of unlearning:

Knowledge Separability Score (KSS):
- Purpose: Measures the overall quality of unlearning by distinguishing between Target Knowledge (to be forgotten) and Non-Target Knowledge (to be retained) across all languages.
- Mechanism: It aggregates a "forgetting score" ( $S_i$ ) for each knowledge instance across languages using both probability-based and generation-based (Semantic Equivalence) methods.
- Components:
  - KSS-ROC: Area Under the Receiver Operating Characteristic Curve, measuring the general separability between forget and retain sets.
  - KSS-PR: Area Under the Precision-Recall Curve, specifically addressing the class imbalance where the forget set is typically much smaller than the retain set.
Knowledge Persistence Score (KPS):
- Purpose: Quantifies the extent to which target knowledge, successfully unlearned in a "base language," persists (remains retrievable) in other languages.
- Mechanism: For a base language $l_1$ and comparison languages $L_2$ , it calculates the fraction of samples where the model has forgotten the knowledge in $l_1$ (Semantic Equivalence = 0) but retains it in $l_2$ (Semantic Equivalence = 1).
- Interpretation: A lower KPS indicates better unlearning performance (less cross-lingual persistence).

Experimental Configuration

The authors evaluated various optimization-based unlearning methods (Gradient Ascent, GAGDR, GAKLR, NPO) and a pruning-based method on the Llama3.1-8B-Instruct model. Experiments were conducted under different forget set ratios (1%, 3%, 5%) and analyzed separately for Training and Hold-out language sets.

3. Key Results

Unlearning Difficulty in Hold-out Languages

Performance Disparity: Unlearning is significantly more difficult in Hold-out languages (Case 1) compared to Training languages (Case 2).
Evidence: KSS-ROC scores were consistently lower for Hold-out languages (e.g., max 0.76 vs. 0.99 in Training languages for the best method), indicating that distinguishing between target and non-target knowledge is harder when the knowledge was acquired via cross-lingual spread rather than direct training.

Impact of Forget Ratio

Degradation: As the forget ratio increases (from 1% to 5%), unlearning performance degrades. The boundary between target and non-target knowledge becomes increasingly obscure, making it difficult for models to distinguish what to forget.
Persistence: KPS scores increased significantly with higher forget ratios, showing that larger forget sets lead to more severe cross-lingual persistence.

Method-Specific Findings

Optimization vs. Pruning: While pruning-based methods achieved high KSS-ROC scores (good global separability), they exhibited disproportionately low KSS-PR scores. Visualization of forgetting scores ( $S_i$ ) revealed that pruning assigned high forgetting scores to a significant amount of non-target knowledge, leading to an overlap in distributions that optimization-based methods avoided.
Persistence Across Languages: Even with successful unlearning in training languages, target knowledge persisted in Hold-out languages (KPS > 0 for all methods), confirming the risk of cross-lingual information leakage.

4. Key Contributions

Comprehensive MMU Scenario: Defined knowledge in MMU as instances expressed across multiple languages, acquired via both direct memorization and cross-lingual spread.
Large-Scale Dataset: Constructed a 3,800-instance multilingual parallel QA dataset spanning 10 languages with rigorous quality control.
Novel Evaluation Metrics: Proposed KSS and KPS to evaluate unlearning quality holistically and measure cross-lingual consistency, respectively.
Empirical Insights: Demonstrated that existing language-wise evaluation protocols are insufficient and that unlearning performance varies significantly between training and hold-out languages.

5. Significance and Limitations

Significance:
The paper argues that reliable MMU evaluation requires metrics that verify information inaccessibility consistently across all languages. The proposed KSS and KPS metrics provide a new paradigm for evaluating MMU, uncovering unique phenomena such as the difficulty of unlearning in hold-out languages and the specific failure modes of pruning-based methods in multilingual settings.

Limitations:

Language Diversity: The hold-out languages were restricted to the Indo-European family (Afrikaans and Spanish), limiting the generalizability of findings across different linguistic structures.
Model Scale: Experiments were limited to an 8B-parameter model due to computational constraints; larger models may exhibit different forgetting dynamics.
Metric Scope: While KSS and KPS address knowledge-wise measurement, the authors acknowledge potential for alternative or more diverse evaluation methodologies.

The authors conclude that future MMU research must explicitly consider multilingual characteristics and aim to unlearn knowledge across languages rather than treating languages as isolated entities.