Technical Summary: De-attribute to Forget for LLM Unlearning (DareU)

1. Problem Statement

The rapid deployment of Large Language Models (LLMs) has intensified concerns regarding the use of inappropriate training data, including intellectual property infringement, private information, and harmful content. Regulations such as the GDPR and CCPA enforce a "right to be forgotten," requiring the removal of specific data influences from trained models. While retraining a model from scratch is the ideal solution, it is computationally prohibitive given the scale of modern LLMs and datasets.

Existing machine unlearning approaches primarily rely on prediction loss-based optimization, typically employing Gradient Ascent (GA) to maximize the loss on the "forget set" (the data to be removed). The paper identifies two critical limitations in these methods:

1. Over-forgetting and Model Collapse: Naively maximizing loss or minimizing log-likelihood often leads to incoherent, gibberish generations rather than a functional model that simply ignores the forget set.
2. Inconsistent Optimization Targets: The target loss value for a perfectly unlearned model varies across individual data samples (as shown by the varying log-likelihoods of retrained models). Consequently, maximizing loss does not provide a consistent, precise objective, often degrading performance on the "retain set" (the data that should be preserved).

The central question posed is whether a more precise optimization objective exists that avoids over-forgetting while effectively removing data influence.

2. Methodology: DareU Framework

The authors propose DareU (Data Attribution-based REinforcement Learning for Unlearning), the first framework to frame LLM unlearning as a data de-attribution problem rather than a prediction loss minimization problem.

Core Concept: Data De-attribution

Instead of maximizing prediction loss, DareU aims to reduce the attribution score of the LLM's generated responses to the owners of the forget data.

• Attribution Function: A function $DA(i, y_d)$ assigns a normalized score $\in [0, 1]$ indicating the likelihood that data owner $i$'s dataset influenced the generation $y_d$.
• Objective: For a forget set $D_F$ belonging to owners $F$, the goal is to minimize the expected attribution score:
  $$\min_{\theta} \mathbb{E}_{i \in F} [\mathbb{E}_{d \sim D_i} (DA(i, y_d))]$$
  Ideally, after unlearning, the attribution score for forget data should be near 0, while the model remains attributable to the retain set owners. This provides a consistent target value (0) for all samples and avoids favoring collapsed, gibberish outputs.

Implementation via Reinforcement Learning (PPO)

Since attribution functions provide scalar feedback, they serve naturally as reward models for Reinforcement Learning (RL). The authors employ Proximal Policy Optimization (PPO) to align the LLM:

1. Reward Signal: The reward is derived from the attribution score. To improve stability, the score is transformed via a log-based function $\chi$ to map probabilities to the range $[-1, 0]$, where lower attribution scores yield higher rewards.
2. Efficient Reward Computation: To avoid the high computational cost of invoking an attribution function for every token, the attribution score is computed every $\kappa$ tokens and broadcast to the intervening tokens. A prefix-averaged base reward is calculated to reduce variance.
3. Distillation Regularization: To preserve model utility on the retain set ($D_R$), a distillation regularization term is added. This term minimizes the KL divergence between the current unlearned policy and the original policy on the retain set, preventing catastrophic forgetting of non-forget data.
4. Attribution Classifier ($D_{Ac}$): Recognizing that ideal attribution functions (e.g., Leave-One-Out, Shapley values) are computationally expensive, the authors train a lightweight LLM classifier (using LoRA) to predict the data owner of a generated response. This classifier is trained offline before unlearning and frozen during the PPO process.

3. Key Contributions

1. Novel Optimization Objective: The paper introduces data de-attribution as a precise objective for LLM unlearning, addressing the inconsistency and instability of prediction loss-based methods.
2. DareU Framework: The authors propose the first unlearning framework utilizing data attribution rewards within a PPO setting. It effectively balances the trade-off between forget quality (removing influence) and model utility (preserving performance on retain data).
3. Empirical Validation: Extensive experiments demonstrate that DareU outperforms existing baselines (including GA, GDiff, SCRUB, and preference optimization methods like IDK and NPO) in achieving effective unlearning without significant degradation of model utility or generation of incoherent text.

4. Experimental Results

The framework was evaluated on two benchmarks: TOFU (Question Answering) and ArXiv (Text Completion), using Llama2-7B and Qwen3-8B models.

• Unlearning Effectiveness: DareU achieved the closest DF (Forget Set) ROUGE scores to the "Retraining" gold standard across both datasets, indicating superior forget quality.
• Model Utility: Unlike Gradient Ascent (GA) and GDiff, which often suffered from over-forgetting (low DR ROUGE), DareU maintained high ROUGE scores on the retain set ($D_R$) and test set ($D_{test}$).
• Balance (ToW Score): DareU achieved the highest Tug-of-War (ToW) score, a metric aggregating the gaps between the unlearned model and the retrained model across forget, retain, and test sets. This confirms DareU's ability to balance forget quality and utility better than any baseline.
• Robustness:
  • Gibberish Reduction: DareU produced significantly fewer incoherent/gibberish responses compared to GA and SCRUB.
  • Adversarial Robustness: The method remained effective against paraphrased forget queries and semantic overlaps between forget and retain sets.
  • Classifier Accuracy: The framework remained robust even when the attribution classifier's accuracy was artificially reduced.
• Efficiency: While DareU incurs higher computational costs than simple prediction loss methods (due to RL generation), it is significantly more efficient than full retraining and comparable to second-order optimization methods like SCuReNewton.

5. Significance and Claims

The paper claims that DareU offers a more precise and stable optimization perspective for LLM unlearning. By shifting the focus from "maximizing error" to "zeroing out attribution," the method avoids the pitfalls of model collapse and inconsistent targets inherent in gradient ascent approaches.

The authors emphasize that while they utilized a lightweight classifier as an efficient approximation, the core contribution is the de-attribution objective itself. They assert that this approach provides a consistent target (score of 0) for all forget samples, which is theoretically superior to the variable targets of loss-based methods.

Limitations Acknowledged:

• Computational Cost: DareU is more computationally expensive than simple fine-tuning or GA, though it offers a justified trade-off for better performance.
• Dependency on Attribution: The method requires access to the training data (or a subset) to train the attribution classifier, which may limit its application to pre-training data removal where such data is inaccessible.
• Approximation: The use of a classifier is an approximation; future work could integrate more accurate or efficient attribution functions to further improve performance.

In conclusion, the paper positions DareU as a state-of-the-art solution that effectively removes data influence while preserving the utility of the LLM, adhering to the "right to be forgotten" without requiring prohibitive retraining costs.