Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Technical Summary: Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Problem Statement
Data unlearning in diffusion models aims to remove the influence of specific training samples (the "forget set") from a pre-trained model while preserving the utility of the remaining data (the "retain set"). Current fine-tuning approaches, such as Gradient Ascent (GA) or Negative Preference Optimization (NPO), typically rely on loss maximization over the forget set. However, these methods often suffer from two critical issues:

1. Incomplete Forgetting: The model may fail to fully erase the specific instance.
2. Quality Degradation: Aggressive unlearning often leads to noisy, aesthetically poor, or distorted generations for the retain set.

Existing methods generally treat all diffusion time steps and frequency components uniformly during the unlearning process. This paper argues that such uniform treatment ignores the inherent dynamics of diffusion models, where different stages of the reverse process (from noise to data) and different frequency components contribute unequally to semantic identity, instance-level memorization, and fine-grained detail.

Methodology
The authors propose a Time- and Frequency-Selective Unlearning framework that targets specific regions of the diffusion process rather than applying uniform updates.

1. Time-Selective Forgetting:

   • Analysis: The paper leverages the theoretical understanding of diffusion dynamics, identifying two critical transition points: Speciation Time ($t_S$) and Collapse Time ($t_C$).
     • Phase I ($t > t_S$): Captures coarse semantics and class identity.
     • Phase II ($t_C < t < t_S$): Involves instance-level attraction and the formation of specific details.
     • Phase III ($t < t_C$): Refines fine-grained details and texture.
   • Strategy:
     • For Unconditional Unlearning, the method focuses on Phase II (middle time steps), where instance-level attraction is strongest, avoiding the degradation of fine details in Phase III.
     • For Conditional Unlearning (text-to-image), the method targets Phase I ($t > t_S$), where conditional guidance dominates the trajectory before speciation occurs.
   • Implementation: A time weighting function $P(t)$ is applied to the loss function, assigning higher probability to specific intervals (e.g., $[t_C, t_S]$ or $[t_S, T]$) while suppressing updates in others. An adaptive method is also proposed to search for the optimal transition point $\tau$ (analogous to $t_S$) based on the guidance magnitude.

2. Frequency-Selective Forgetting:

   • Analysis: Empirical spectral analysis reveals that unlearning often introduces unwanted high-frequency noise, degrading sample quality. The retain set's high-frequency distribution should remain largely unchanged.
   • Strategy: A low-pass filter is applied to the loss residual during unlearning. The gradient signal is projected to remove high-frequency components of the forget set, while the input to the denoiser remains the natural noisy sample.
   • Implementation: Using the Fast Fourier Transform (FFT), the loss residual is masked to exclude frequencies above a cutoff radius $r_t$. This ensures that the unlearning process does not perturb the high-frequency details essential for image quality.

3. Theoretical Justification:

   • The authors formulate a forgetting–preservation utility function $J_\beta(m)$ based on the Kullback-Leibler (KL) divergence between the original and unlearned path measures.
   • They prove that selective unlearning (updating only in region $S$) yields a strictly positive utility gap compared to full unlearning, provided that the selective updates amplify changes in the target region ($\mu \ge 1$) and suppress changes in non-target regions ($\eta \le 1$).

4. Evaluation Metric:

   • The paper introduces $SSCD_{norm}$, an improved metric that normalizes the perturbation distance. Unlike the standard SSCD, which can be artificially low for blurry or degraded images, $SSCD_{norm}$ accounts for the quality of the generated image, ensuring that low dissimilarity is not achieved at the cost of aesthetic degradation.

Key Results
The proposed method was evaluated on CelebA-HQ (unconditional) and Stable Diffusion v1.4 (conditional text-to-image) across various baselines (GA, SISS, DPO, KTO).

• Unconditional Unlearning: Applying time selection (focusing on $t \in [250, 750]$) and frequency selection (low-pass filter with $r_t=0.15$) significantly improved the $SSCD_{norm}$ and Aesthetic Scores compared to baselines. For instance, with SISS, the aesthetic score improved by ~34.8% at $t=250$ and ~25.8% at $t=500$, while maintaining or improving forgetting metrics.
• Conditional Unlearning: On Stable Diffusion, the selective approach achieved a higher unlearning success rate and faster convergence than the SISS baseline. It successfully removed memorized prompts while maintaining higher CLIP-IQA quality scores.
• Membership Inference Attacks (MIA): Existing methods showed extreme vulnerability to MIA (relative errors up to $\times 10^7$), indicating over-forgetting. The proposed method substantially mitigated this collateral distortion (reducing errors to $\times 5.76$ and $\times 53.20$), suggesting better preservation of the data manifold.
• Generalization: The method demonstrated effectiveness across different backbones (U-Net and Diffusion Transformer) and concept unlearning scenarios (e.g., nudity detection).

Significance and Claims
The paper claims to provide practical insights for understanding and improving data unlearning in diffusion models by demonstrating that forgetting is not a uniform process. The primary contributions are:

1. Systematic Study: A detailed analysis of how time steps and spectral domains contribute differently to memorization and utility.
2. Simple Framework: A selective unlearning approach that is compatible with existing objectives (GA, SISS, DPO, KTO) and requires minimal architectural changes.
3. Quality Preservation: The ability to achieve high unlearning success rates without the severe quality degradation typically associated with gradient-based unlearning.
4. Improved Evaluation: The introduction of $SSCD_{norm}$ to prevent the evaluation of unlearning methods that rely on image degradation as a proxy for forgetting.

The authors modestly note that while their method effectively identifies optimal regions based on existing theoretical bounds, it does not yet provide a fully automated method for discovering these optimal regions for every specific dataset without prior knowledge or adaptive search overhead. Nevertheless, the work highlights that targeted forgetting in specific time and frequency windows is crucial for balancing the trade-off between privacy (unlearning) and utility (generation quality).