💬 NLP

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

This paper reveals a significant gap between standard clean-query metrics and adversarial robustness in machine unlearning, demonstrating that while fine-tuning methods appear to successfully forget data under conventional evaluation, targeted information remains highly recoverable through strategic prompting, thereby highlighting the critical need for adversarial stress-testing to ensure true unlearning.

Original authors: Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni, Varad Ingale, Tuhina Tripathi, Abhishek Lalwani, Somya Chatterjee, Sadid Hasan

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni, Varad Ingale, Tuhina Tripathi, Abhishek Lalwani, Somya Chatterjee, Sadid Hasan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a library where every book contains the sum of human knowledge, but some pages hold secrets that must be erased: private addresses, outdated medical advice, or proprietary formulas. In the digital world, artificial intelligence systems act as these vast libraries, trained on enormous collections of text. When a user asks a computer to "forget" a specific piece of information, the goal is to remove that data so thoroughly that the machine can no longer recall it, while still remembering everything else. This process, known as machine unlearning, is vital for privacy and safety. However, a lingering question has haunted researchers: if a computer says it has forgotten something, can it truly be trusted? Or is the information merely hidden, waiting to be dug up by a clever question?

A team of researchers from the University of Massachusetts Amherst and Microsoft Corporation set out to answer this by testing the limits of how well these digital libraries can actually forget. They focused on a specific type of artificial intelligence model, one that is trained to follow instructions and answer questions. The team wanted to see if the standard tests used to measure forgetting were enough, or if they were missing a crucial weakness. To do this, they created a controlled experiment using a dataset of made-up author biographies. They taught the model to know these fake authors and then tried to make it forget specific ones. They tested two main ways of doing this: one method that tried to suppress the memory by changing how the model was asked a question, and another that actually rewired the model's internal connections to erase the memory.

The researchers first ran the standard tests, which involve asking the model simple, direct questions about the information it was supposed to forget. Under these calm, non-adversarial conditions, several of the methods appeared to work perfectly. The models scored very high marks, suggesting they had successfully erased the targeted facts. One method, in particular, seemed to have forgotten the information almost completely, with a score indicating near-total success. It looked as though the data was gone for good.

But the team did not stop there. They suspected that a model might be able to resist a simple question while still being tricked by a more complex one. To test this, they launched a series of strategic attacks. Instead of asking directly, they used a "hacker" program to generate hundreds of tricky prompts designed to bypass the model's defenses. These prompts included role-playing scenarios where the model was asked to pretend to be an expert, instructions that tried to override its safety rules, and questions phrased in hypothetical or indirect ways. They also tried asking the same questions in different languages to see if the model would slip up when the language changed.

The results revealed a startling gap between what the standard tests showed and what was actually happening. While the models had passed the simple tests with flying colors, they failed miserably when faced with the strategic attacks. The researchers found that even the models that claimed to have forgotten the information could be coaxed into revealing it again. In fact, when attacked with these clever prompts, the models gave away the secret information in more than seventy-two percent of the attempts. For some methods, this failure rate was nearly as high as the model that had never been taught to forget anything at all. The information was not gone; it was just lying dormant, waiting for the right key to unlock it.

The team also discovered that the way the question was asked mattered immensely. Direct questions were sometimes easier to resist, but prompts that used authority figures, asked the model to complete a sentence, or framed the request as a hypothetical scenario were far more effective at breaking the model's memory. Interestingly, when the researchers simply translated the questions into other languages like German, Spanish, or Japanese without adding any tricky framing, the models held up much better, leaking the information less than three percent of the time. This suggests that the vulnerability is not just about language, but about the specific structure of the trick used to ask the question.

To ensure their findings were accurate, the researchers had a human expert review a small sample of the model's responses. They found that the automated system used to judge the leaks was mostly correct, agreeing with human judgment in seven out of ten cases. While the automated system occasionally made mistakes—sometimes flagging a wrong answer as a leak or missing a correct one—it provided a reliable signal that the information was indeed being recovered. This confirmed that the high success rate of the attacks was real and not just a glitch in the testing software.

The study concludes that the current way of testing if a machine has forgotten something is insufficient. A model can look perfect on a standard report card while still being dangerously vulnerable to a determined adversary. The researchers argue that we cannot rely on simple, clean questions to prove that data has been removed. Instead, we must stress-test these systems with the same kind of clever, strategic questioning that a real-world attacker might use. Until we can prove that a model cannot be tricked into revealing its secrets, we cannot be certain that it has truly forgotten them. The gap between appearing to forget and actually forgetting remains wide, and bridging it will require new ways of testing and building these powerful digital minds.

Technical Summary: Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

Problem Statement

Machine unlearning aims to remove the influence of specific training data from a Large Language Model (LLM) while preserving its performance on retained data. A central assumption in current research is that strong performance on standard benchmark metrics (e.g., high Forget Quality) indicates that targeted knowledge has become inaccessible. However, existing evaluations primarily rely on clean, non-adversarial queries. This leaves a critical gap: it remains unclear whether information that appears "forgotten" under standard testing can be recovered through strategic prompting, such as role-playing, prompt injection, or rephrasing. Furthermore, prior work often evaluates prompt-based and fine-tuning-based unlearning methods under disparate experimental settings (different models, datasets, and metrics), making direct comparisons of their effectiveness difficult.

Methodology

The authors propose a unified evaluation framework to address these gaps, utilizing the TOFU (Task of Fictitious Unlearning) dataset and the Llama-3.2-3B-Instruct model. The methodology proceeds in three stages:

Unified Benchmarking:
- Setup: A base model is fine-tuned on the complete TOFU dataset (200 fictitious authors). This checkpoint serves as the common starting point for all unlearning experiments.
- Methods Evaluated: Six methods are compared across 1%, 5%, and 10% forget splits:
  - Prompt-based: In-Context Unlearning (ICUL) and Soft Prompting for Unlearning (SPUL). These do not modify model weights.
  - Fine-tuning-based: Negative Preference Optimization (NPO), Simple NPO (SimNPO), Representation Misdirection for Unlearning (RMU), and Alternate Preference Optimization (AltPO). These modify model parameters or internal representations.
- Metrics: Evaluation uses the OpenUnlearning framework, measuring Forget Quality (FQ), Model Utility (MU), Forget Truth Ratio (FTR), and Privacy Leakage (PrivLeak).
Adversarial Robustness Evaluation:
- Scope: Focused on the four fine-tuning-based methods (which demonstrated higher clean-query FQ) on the 1% forget split.
- Attack Generation: An adversarial pipeline uses a "hacker" model (Llama-3.1-8B-Instruct) to generate and iteratively refine prompts.
- Attack Suites: Eight attack types across four categories are employed:
  - Role-playing: Persona framing.
  - Prompt Injection: Authority framing and system instructions.
  - Rephrasing: Direct, hypothetical, multiple-choice, and structured chain-of-thought formulations.
  - Knowledge Routing: Completion-style prompts.
- Evaluation Metric: An Attack Success Rate (ASR) is introduced. A "judge" model (QLoRA-tuned TOFU model) assigns a leakage score (0–1) to the victim model's response. An attack is successful if the leakage score exceeds 0.2. ASR is the fraction of successful attacks.
Validation and Multilingual Probing:
- Judge Validation: The LLM judge's leakage scores are correlated with ROUGE-L and validated via a manual audit of 10 adversarial responses.
- Multilingual Testing: The 1% forget set is translated into seven languages (German, French, Spanish, Hindi, Chinese, Arabic, Japanese) to test for cross-lingual recovery under clean queries.

Key Contributions

Unified Cross-Paradigm Evaluation: The paper provides a direct comparison of prompt-based and fine-tuning-based unlearning methods under a shared framework. It reveals that author-reported metrics for prompt-based methods (e.g., token-level accuracy) can substantially overestimate effectiveness, as these methods degrade to near-zero Forget Quality as the forget set size increases, whereas fine-tuning methods maintain high FQ.
Introduction of Attack Success Rate (ASR): The authors introduce ASR as a metric to quantify the recoverability of "forgotten" information under adversarial conditions. They demonstrate that high Forget Quality does not correlate with resistance to strategic prompting.
Adversarial Stress-Testing: The study reveals a substantial gap between clean-query forgetting and adversarial robustness. While fine-tuning methods achieve high FQ (0.91–0.99), they remain highly vulnerable to adversarial recovery, with ASRs ranging from 72.8% to 84.3% (compared to 87.5% for the unprotected base model).
Multilingual vs. Adversarial Comparison: The paper contrasts adversarial English prompting with clean multilingual reformulations. While adversarial prompts yield high leakage, clean translations into seven languages result in a significantly lower measured leakage rate (2.95%), suggesting the primary robustness gap lies in adversarial prompting rather than cross-lingual queries.

Results

Clean-Query Performance: Fine-tuning methods (NPO, SimNPO, AltPO) consistently achieve high Forget Quality (>0.91) and stable Model Utility. Prompt-based methods (SPUL, ICUL) show a sharp decline in Forget Quality as the forget set size increases (dropping to ~0.001 at 10%).
Adversarial Vulnerability: Despite high FQ, the four fine-tuning methods exhibit high ASRs:
- NPO: 72.8%
- SimNPO: 74.7%
- AltPO: 75.3%
- RMU: 84.3%
- Comparison: These values are only 3.2–14.7 percentage points lower than the unprotected base model (87.5%).
Attack Effectiveness: Prompt-injection attacks (Authority, System Injection) were the most effective (avg ASR ~86.6%). Rephrasing attacks were less effective overall but varied significantly; Multiple Choice prompts were the most robust against recovery (ASR 45–70%), while Direct and Hypothetical prompts remained highly effective.
Metric Validation: The LLM judge leakage scores showed a strong Pearson correlation with ROUGE-L ( $r = 0.924$ ). A manual audit of 10 cases found agreement between the automated ASR decision and human factual assessment in 7 cases, indicating ASR is a useful but imperfect signal.
Multilingual Leakage: Clean multilingual queries resulted in a 2.95% leak rate, significantly lower than adversarial English attacks. German showed the highest leakage (10%), while Arabic, Hindi, and Japanese showed 0%.

Significance and Claims

The paper claims that strong performance on standard clean-query metrics is insufficient to establish robustness after unlearning. The primary significance lies in demonstrating that information appearing "forgotten" under standard evaluation can be easily recovered through strategic prompting.

The authors argue that current unlearning benchmarks provide a false sense of security. They motivate the inclusion of adversarial stress-testing as a complementary component of unlearning evaluation. The findings suggest that while fine-tuning-based methods outperform prompt-based methods in standard metrics, neither paradigm currently offers robust resistance to targeted knowledge recovery via adversarial prompts. The paper concludes that future evaluations must move beyond clean queries to assess whether unlearning behavior persists under strategic reformulation.

Note: The paper explicitly limits its scope to the Llama-3.2-3B-Instruct model and the synthetic TOFU dataset, acknowledging that generalization to larger models, different architectures, or real-world sensitive data requires further investigation.