🤖 machine learning

Leak@ $k$ : Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding

This paper reveals that existing LLM unlearning methods fail to achieve true forgetting under probabilistic decoding, introducing the \texttt{leak@ $k$ } metric to quantify this vulnerability and proposing the \texttt{RULE} algorithm to effectively mitigate knowledge leakage across multiple benchmarks.

Original authors: Hadi Reisizadeh, Jiajun Ruan, Yiwei Chen, Soumyadeep Pal, Sijia Liu, Mingyi Hong

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hadi Reisizadeh, Jiajun Ruan, Yiwei Chen, Soumyadeep Pal, Sijia Liu, Mingyi Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Amnesia" Illusion

Imagine you hire a librarian (the AI) and ask them to remove a specific, embarrassing book from their collection. You want them to completely forget the story so they never tell it again.

The researchers in this paper discovered a shocking truth: The librarian isn't actually forgetting the story.

When you ask the librarian a question, they usually give you a standard, safe answer (like a robot reading a script). This is called Greedy Decoding. Under these conditions, the librarian looks like they have successfully forgotten the book.

However, if you ask the librarian to "improvise" or "be creative" (which is how real-world AI actually works, called Probabilistic Decoding), the librarian suddenly remembers the story and tells it to you, often word-for-word. The "forgetting" was just an illusion caused by how we were testing them.

The Problem: The "Magic 8-Ball" vs. The "Crystal Ball"

To understand the paper, we need to look at how AI makes decisions:

Greedy Decoding (The Crystal Ball): Imagine the AI always picks the single most obvious, safe word next. It's like a crystal ball that only shows you the most likely future. In this mode, the AI successfully hides the secret.
Probabilistic Decoding (The Magic 8-Ball): In the real world, AI doesn't just pick the most likely word; it picks from a list of good options, sometimes taking a riskier path to be more creative or human-like. This is like shaking a Magic 8-Ball.
- The Discovery: The paper found that while the AI hides the secret when looking through the Crystal Ball, it spills the secret when you shake the Magic 8-Ball enough times. If you ask the same question 64 times with different "shakes" (randomness), the secret eventually leaks out.

The New Tool: "Leak@k"

The authors realized that current tests were like checking a dam only when the river is calm. They needed a way to test the dam during a storm.

They invented a new measuring stick called Leak@k.

The Analogy: Imagine you are trying to see if a sieve has holes.
- Old way: You pour one cup of water through the sieve. No water comes out? Great, no holes! (This is the old "Greedy" test).
- Leak@k way: You pour k cups of water through the sieve. Even if the first cup doesn't leak, the 10th or 50th cup might find a tiny hole and drip through.
What it measures: It calculates the probability that at least one of those many attempts will reveal the secret information. The paper shows that for almost all current "unlearning" methods, as you increase the number of cups (k), the water (secrets) eventually leaks out.

The Evidence: The "Bus to Hell" Example

The paper provides a funny but serious example using a dataset about news articles (MUSE).

The Secret: A bus route number that was changed from 666 (Hell) to 669.
The Test:
- Greedy Mode: The AI says, "I don't know that route." (Success!)
- Probabilistic Mode (64 tries): The AI eventually says, "The new route number is 669." (Failure!)

This happened across three different major tests (TOFU, MUSE, and WMDP). Whether the secret was a fake author's name, a news fact, or dangerous biological knowledge, the "forgetting" failed as soon as the AI was allowed to be creative.

The Solution: RULE (Robust Unlearning)

Since the old methods were like trying to erase a tattoo with a wet paper towel (it looks gone, but the ink is still there), the authors created a new method called RULE.

How it works: Instead of just telling the AI "Forget this specific sentence," RULE plays a game of "Whac-A-Mole."
1. It asks the AI the question many times to see how it tries to leak the secret.
2. It catches the AI when it almost slips up.
3. It then teaches the AI to forget those specific slips too.
4. It repeats this process, getting stricter every time.
The Result: With RULE, even if you shake the Magic 8-Ball 128 times, the secret stays hidden. The AI truly forgets, not just when it's being careful, but even when it's being creative.

Summary

The Issue: Current AI "unlearning" methods are fake. They only work when the AI is forced to be boring and predictable.
The Reality: When the AI is allowed to be creative (which is how we use it), it remembers the things we told it to forget.
The Fix: The authors built a new test (Leak@k) to catch this cheating and a new training method (RULE) to actually make the AI forget, even under pressure.

The paper concludes that we cannot trust current AI safety measures until we test them with this new, stricter "Leak@k" standard.

Technical Summary: Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding

Problem Statement

Large Language Models (LLMs) are trained on vast web datasets containing private, toxic, illegal, or copyrighted content. Machine unlearning aims to remove this specific knowledge while preserving the model's general utility. However, current evaluation protocols for unlearning rely almost exclusively on deterministic (greedy) decoding (temperature $T=0$ , top- $p=0$ ). In this setting, the model selects the single most probable token at each step.

The authors identify a critical gap: while unlearning methods often appear successful under greedy decoding, they fail to achieve "true forgetting" under probabilistic decoding (e.g., $T > 0$ or top- $p > 0$ ), which is standard in real-world deployments like conversational agents and code generation. Under probabilistic sampling, sensitive information that was suppressed in the greedy path can resurface. Once an initial sensitive token is sampled, the model's conditional distribution often shifts, causing subsequent sensitive tokens to be generated with high confidence. This creates a security blind spot where "unlearned" models can still leak private or harmful knowledge when queried multiple times or with stochastic parameters.

Methodology

1. The Leak@k Meta-Metric

To rigorously quantify this vulnerability, the paper introduces leak@k, a distributional meta-metric. Unlike prior evaluations that measure leakage per single generation or average it across generations, leak@k quantifies the likelihood that at least one sensitive response appears among $k$ independent samples generated from the same prompt under probabilistic decoding.

Definition: For a prompt $q$ , let $S_j$ be a correctness score (e.g., ROUGE-L, Entailment Score, Accuracy) for the $j$ -th generated response $f(q; \theta)$ against the ground truth. The metric is defined as the expected maximum score over $k$ samples:
$\text{leak@k} := \mathbb{E}\left[\max_{1 \le j \le k} S_j\right]$
Estimation: The authors propose an unbiased estimator, $\widehat{\text{leak@k}}$ , which generates $n \ge k$ samples and computes the maximum score over all possible subsets of size $k$ to reduce variance compared to a naive "worst-of- $k$ " approach.
Decay Analysis: To characterize how leakage evolves as the sampling budget grows, the authors fit a log-log model: $L_k = 1 - (1 - L_1)k^{-\gamma}$ . Here, $L_1$ represents early-stage leakage, and $\gamma$ represents the decay rate of the remaining non-leaked mass. A successful unlearning method should exhibit small $L_1$ and small $\gamma$ .

2. The RULE Algorithm

To address the failure of static unlearning sets under probabilistic decoding, the authors propose Robust Unlearning under LEak@k metric (RULE). This is an iterative, meta-algorithm that dynamically augments the forget set:

Multi-sample Generation: Using the current model and probabilistic decoding, generate multiple candidate responses for queries in the forget set.
Leakage Detection: Filter these responses using a core metric (e.g., Entailment Score). Responses that successfully reveal sensitive information (exceeding a threshold) are identified.
Dataset Augmentation: These "leaking" responses are added to the forget set, effectively teaching the model to forget not just the original ground truth, but also the specific variations and paraphrases it tends to generate under stochastic conditions.
Optimization: The model is retrained on this augmented forget set. This process repeats for several iterations.

Key Contributions

Introduction of Leak@k: A new meta-metric that shifts the evaluation paradigm from deterministic single-response checks to probabilistic worst-case analysis across multiple generations.
Systematic Empirical Study: The first large-scale evaluation of unlearning reliability across three major benchmarks (TOFU, MUSE-News, and WMDP) using various decoding strategies ( $T, p$ ) and sampling budgets ( $k$ ).
Demonstration of Failure: The study reveals that state-of-the-art (SOTA) methods (including GradDiff, NPO, SimNPO, RMU, and BLUR-NPO) provide only an "illusion of forgetting." As $k$ increases, leak@k rises sharply for all existing methods, indicating that sensitive knowledge reliably resurfaces under realistic sampling.
Proposal of RULE: A dynamic training algorithm that significantly reduces information leakage. On the TOFU benchmark, RULE achieves near-zero leak@k even for $k=128$ , outperforming all baselines. On MUSE-News, it outperforms SOTA methods across most sampling budgets.

Results

TOFU Benchmark: Using LLaMA-3.2-1B, the authors show that while greedy decoding ( $T=0, p=0$ ) suggests successful unlearning for methods like NPO and BLUR-NPO, probabilistic decoding ( $T=1.0, p=1.0$ ) causes leak@k (measured by Entailment Score) to approach 1.0 as $k$ increases. The top- $p$ parameter was found to be the primary driver of leakage, more so than temperature.
MUSE-News Benchmark: Similar trends were observed with LLaMA2-7B. Existing methods transition from safe to leaking outputs as $p$ increases or $k$ grows. RULE-NPO demonstrated superior performance in suppressing leakage growth rates ( $\gamma$ ).
WMDP Benchmark: Using Zephyr-7B-beta, the study highlighted that accuracy-based metrics can be misleading. While some methods (like NPO) achieved low accuracy on multiple-choice questions, they often generated gibberish or "over-forgot" (selecting a refusal option), whereas others (like RMU) showed clear leakage trends under high randomness when evaluated via LLM-as-Judge on free-form generations.
Utility Preservation: Across all benchmarks, the proposed methods (including RULE) maintained stable performance on the retain set, indicating that robust forgetting does not necessarily come at the cost of general model utility.

Significance and Claims

The paper claims that current unlearning evaluations are fundamentally flawed because they ignore the probabilistic nature of LLMs, creating a false sense of security. The authors argue that unlearning must be evaluated under the same probabilistic conditions used in deployment to ensure safety.

The significance of this work lies in:

Exposing a Critical Gap: Demonstrating that "successful" unlearning under greedy decoding is insufficient for regulatory compliance and ethical AI, as sensitive data can be recovered via repeated sampling.
Providing a Robust Evaluation Framework: Leak@k offers a standardized, distributional metric to compare unlearning methods based on their worst-case leakage potential rather than average-case performance.
Offering a Solution: RULE demonstrates that it is possible to achieve robust unlearning by iteratively refining the forget set based on the model's own probabilistic failures, moving beyond static training sets.

The authors conclude that without such robust evaluation and training methods, the deployment of unlearned LLMs in high-stakes environments remains risky, as adversarial users can exploit probabilistic decoding to extract "forgotten" information.