PrivUn: Unveiling Latent Ripple Effects and Shallow Forgetting in Privacy Unlearning

Technical Summary: PrivUn: Unveiling Ripple Effects and Shallow Forgetting in Privacy Unlearning

1. Problem Statement

As Large Language Models (LLMs) increasingly memorize Personally Identifiable Information (PII) during training, machine unlearning has emerged as a vital defense. However, the authors identify two critical gaps in the current landscape:

• Evaluation Gap: Existing benchmarks (e.g., TOFU, MUSE) primarily test against passive attackers (those who merely observe model outputs). They fail to account for active attackers who can use In-Context Learning (ICL) or fine-tuning to "relearn" and extract forgotten data.
• Mechanistic Gap: There is a lack of understanding regarding how PII—which lacks the explicit semantic/logical structure of general knowledge (like knowledge graphs)—is entangled and stored within a model's optimization landscape.

2. Methodology

The paper introduces PrivUn, a comprehensive evaluation framework and a set of new analytical metrics to study the robustness of PII unlearning.

A. The PrivUn Benchmark:
The framework assesses unlearning across a three-tier attack hierarchy:

1. P1 (Direct Retrieval): Testing if the model can still answer direct questions about the PII.
2. P2 (ICL Recovery): Testing if a few-shot prompt can trigger the model to reveal the forgotten data.
3. P3 (Fine-tuning Restoration): Testing if an attacker can recover the data by fine-tuning the model on a small set of original samples.

B. Quantitative Metrics:

• Forgetting Score (FS): Measures the divergence in log-probabilities of PII sequences between the target and unlearned models.
• Association Score (AS): Quantifies data entanglement via three lenses: Gradient-based (optimization similarity), Representation-based (hidden state similarity), and Graph-based (social/relational proximity).
• Forgetting Depth Assessment: Uses Layer-wise Centered Kernel Alignment (CKA) to track how much the model's internal representations change across different layers.

3. Key Findings

Through a systematic study of 19 unlearning methods, the authors uncover two fundamental phenomena:

• Finding 1: Gradient-Driven Ripple Effects: Unlearning a specific piece of data causes a "ripple effect" where unknown, related data is also forgotten. Crucially, this is not driven by human-interpretable semantic relations (like social networks) but by latent gradient-based associations. Samples that share similar parameter update directions during optimization are more likely to be co-forgotten.
• Finding 2: Shallow Forgetting: Most current methods (especially label-based ones) only modify the model's output distribution (the "surface"). CKA analysis reveals that while the final layers might change, the intermediate representations remain largely untouched. This allows active attackers to easily "reconstruct" the PII through fine-tuning.

4. Proposed Strategies (Solutions)

To move from "shallow" to "deep" forgetting, the authors propose two novel strategies:

1. Association-Aware Core-set Selection: Instead of selecting random data to forget, this method selects a "core set" of samples with the highest gradient similarity to the target forget set. This leverages the ripple effect to achieve high forgetting rates with minimal data (e.g., using only 10% of the data to match the effect of 50%).
2. Representation Anchoring Unlearning (RAU): To solve shallow forgetting without destroying model utility, RAU intervenes in the hidden layers. Instead of pushing representations toward random noise (which kills utility), it anchors them to the base model's states at specific intermediate depths (e.g., layer 20), ensuring deep erasure while preserving general capabilities.

5. Results and Significance

• Results: The proposed strategies significantly outperform existing methods. The core-set selection achieves a 32.19% P3 recovery rate using only 10% data. RAU reduces the P3 recovery rate to 30.39% while maintaining much higher utility (MMLU/QA scores) than previous representation-based methods like RMU.
• Significance: This paper shifts the paradigm of privacy unlearning from output-focused (changing what the model says) to representation-focused (changing what the model knows internally). It provides the community with a more rigorous standard for "true" unlearning and offers a mechanistic roadmap for building models that are robust against sophisticated, active privacy attacks.