Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs

Technical Summary: Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs

Problem Statement
Machine unlearning (MU) for Large Language Models (LLMs) aims to remove specific undesirable data or knowledge (e.g., personal identifiers, copyrighted content, or harmful capabilities) while preserving model utility on standard tasks. While existing approximate unlearning methods (such as gradient ascent, preference optimization, and representation editing) are designed to suppress the influence of a "forget set," their effectiveness is often evaluated solely on whether the model fails to answer forget-relevant queries.

This paper identifies a previously unexplored vulnerability: unlearning trace detection. The authors posit that unlearning leaves behind persistent "fingerprints" in both model behavior and internal representations. These traces allow an adversary to determine whether a model has undergone unlearning purely by observing its input-output behavior, even when the model is queried with forget-irrelevant prompts (inputs unrelated to the erased knowledge). If an adversary can detect that a model has been unlearned, they may strategically invest resources to reverse-engineer the forgotten information (e.g., via jailbreaking or relearning attacks), thereby undermining the privacy and safety guarantees of the unlearning process.

Methodology
The study frames unlearning trace detection as a supervised classification task. The core hypothesis is that unlearned models exhibit distinct behavioral and representational shifts compared to their original counterparts.

1. Experimental Setup:

   • Models: Four instruction-tuned LLMs were used: Zephyr-7B, LLaMA-3.1-8B, Qwen2.5-14B, and Yi-34B.
   • Unlearning Methods: Two state-of-the-art approaches were evaluated:
     • RMU (Representation Reengineering): Modifies intermediate activations of forget-set samples to a fixed random vector.
     • NPO (Negative Preference Optimization): Treats forget-set samples as negative examples in a preference optimization framework to push the model away from pre-unlearning behavior.
   • Datasets: The forget set consisted of 3,668 hazardous biosecurity/cybersecurity questions from the WMDP benchmark. Forget-irrelevant data was drawn from MMLU (general utility) and UltraChat.
   • Inputs: The study evaluates detection using two types of outputs:
     • Textual Responses: Discrete "hard" outputs generated by the model.
     • Pre-logit Activations: Continuous "soft" activation vectors (hidden states) extracted before the final softmax layer.

2. Detection Pipeline:

   • A lightweight supervised classifier (a two-layer MLP) is trained to distinguish between "original" and "unlearned" models.
   • For textual responses, an LLM2Vec encoder is used to embed the generated text before classification.
   • For activations, the MLP is trained directly on the pre-logit vectors.
   • The classifiers are tested on unseen prompts from WMDP (forget-relevant), MMLU, and UltraChat (forget-irrelevant).

3. Spectral Analysis:

   • To understand the mechanism behind detection, the authors perform Singular Value Decomposition (SVD) on centered activation matrices. They project activations onto the top right singular vectors (SV1) to visualize distributional shifts ("spectral fingerprints") between original and unlearned models.
   • They also employ UMAP (Uniform Manifold Approximation and Projection) to investigate low-dimensional nonlinear manifolds in the activation space.

Key Results

1. High Detectability: Simple supervised classifiers can distinguish unlearned models from originals with high accuracy (>90% in many cases), even when the model is prompted with forget-irrelevant inputs.

   • NPO Models: Exhibit strong detectability. Classifiers trained solely on textual responses achieve near-perfect accuracy (99–100%) across all model sizes and prompt types. This is attributed to the aggressive, global behavioral shifts NPO imposes.
   • RMU Models: Exhibit subtler traces. Text-based detection is less effective on forget-irrelevant prompts (e.g., ~54% for Zephyr-7B on MMLU, close to random guessing). However, classifiers trained on pre-logit activations achieve near-perfect accuracy (>95%) even for RMU models on forget-irrelevant data.

2. Spectral Fingerprints:

   • NPO: Shows pronounced distributional shifts in the final layer activations along the first singular vector (SV1).
   • RMU: Does not show clear shifts in the final layer. However, distinct spectral fingerprints are observed in intermediate layers (specifically the down-projection sublayers of the Feed-Forward Networks, e.g., L7.D_PROJ).
   • Manifold Separation: Despite the lack of linear separation in final activations for RMU, UMAP analysis reveals that original and unlearned activations lie on separable low-dimensional nonlinear manifolds. This explains why neural classifiers (which can learn non-linear boundaries) succeed where linear spectral analysis might appear inconclusive.

3. Model Scale and Generalization:

   • Larger models (e.g., Yi-34B) exhibit stronger detectability than smaller models (e.g., Zephyr-7B).
   • Classifiers trained on pre-logit activations from one model family (e.g., Zephyr) can generalize to detect unlearning in other model families (e.g., Yi, Qwen) with high accuracy, though performance varies depending on the specific model architecture.

4. Forget-Data Detection:

   • Once a model is identified as unlearned, the paper demonstrates that it is possible to detect whether a specific response originated from the "forget domain" (hazardous content) versus general knowledge, achieving >70% accuracy by leveraging statistical shifts in entropy and prediction probabilities.

Significance and Claims
The paper claims to reveal a critical vulnerability in the current paradigm of LLM unlearning. The authors argue that:

• Unlearning is not invisible: The process of removing knowledge leaves measurable, persistent signatures in both the model's output text and its internal activation space.
• Reverse-Engineering Risk: The ability to detect unlearning traces allows adversaries to identify "unlearned" models and potentially focus computational resources on recovering the erased information, effectively bypassing the intended privacy or safety protections.
• Limitations of Current Methods: The study highlights that current approximate unlearning methods, while effective at suppressing specific answers, fail to erase the "fingerprint" of the unlearning process itself.
• Forensic Necessity: The findings suggest that evaluating unlearning requires more than just checking performance on forget-relevant queries; it necessitates a forensic analysis of the model's behavioral and representational traces to ensure true erasure.

The authors conclude that unlearning traces are a fundamental byproduct of current unlearning algorithms, introducing a new risk vector for reverse-engineering forgotten information in high-stakes domains.