Technical Summary: Locality-Aware Continual Unlearning for Diffusion Models

1. Problem Statement

The deployment of text-to-image diffusion models in real-world scenarios necessitates Continual Unlearning (CUL): the sequential removal of specific concepts (e.g., due to privacy, copyright, or safety regulations) while preserving the model's ability to generate all other content.

Existing unlearning methods are primarily designed for single-step (one-shot) deletion. When applied sequentially, these methods exhibit rapid instability, typically collapsing after only 3–5 steps. The authors identify three compounding factors responsible for this failure:

1. Coarse Mapping Targets: Current methods redirect diverse forget prompts to a single fixed anchor (e.g., an empty string or generic concept). This forces unnecessarily large displacements in the model's score field for every prompt, causing cumulative degradation of generative fidelity.
2. Lack of Local Protection: Unlearning induces a "ripple effect," where semantically neighboring concepts (those sharing internal representations) suffer the most collateral damage. Existing global replay strategies distribute regularization uniformly, failing to protect these vulnerable local neighborhoods.
3. Text-Space vs. Model-Space Mismatch: Methods relying on text-space similarity (e.g., CLIP embeddings) do not accurately reflect how the diffusion model internally processes prompts. Prompts close in CLIP space may induce vastly different noise predictions, leading to suboptimal target selection that compounds errors over sequential steps.

2. Methodology: Locality-Aware Continual Unlearning (LACU)

The authors propose LACU, a framework that grounds both target selection and replay in a single, model-aware principle: Score-Prediction Distance. This metric measures the difference in noise predictions between two prompts on shared noisy latent states, effectively quantifying proximity within the model's own denoising landscape.

The framework operates as a multi-objective teacher-student process at each continual step $i$, where the frozen previous model $\hat{\theta}_{i-1}$ acts as the teacher and the updated model $\theta_i$ is the student. The total loss is defined as:
$$\mathcal{L}_{\text{total}} = \lambda_{\text{unlearn}} \mathcal{L}_{\text{unlearn}} + \lambda_{\text{retain}} \mathcal{L}_{\text{retain}} + \lambda_{\text{reg}} \mathcal{L}_{\text{reg}}$$

Key Components:

1. Locality-Aware Target Selection:

   • Instead of mapping all forget prompts to a fixed anchor, LACU generates context-preserving candidate replacements for each specific forget prompt.
   • It selects the candidate with the minimum score-prediction distance to the original forget prompt.
   • Goal: Minimize the perturbation of the learned score field for each step, ensuring updates are as small and targeted as possible.

2. Locality-Aware Replay:

   • The framework identifies retain concepts that are closest to the forget concept in the model's noise-prediction space (using the same score-prediction distance metric).
   • These nearest neighbors are replayed as a local functional regularizer.
   • Goal: Directly shield the most vulnerable semantic neighborhood from the ripple effect, rather than relying on global replay which under-invests in these fragile regions.

3. Stabilization Mechanisms:

   • Teacher-Student Distillation: The student mimics the teacher's behavior on both the selected mapping targets and the local replay neighbors.
   • Lightweight $\ell_2$ Regularization: A penalty term ($\|\theta_i - \hat{\theta}_{i-1}\|_2^2$) limits cumulative parameter drift over sequential steps.

3. Key Contributions

• Diagnosis of Sequential Instability: The paper traces the collapse of existing methods in CUL to coarse mapping, lack of local protection, and the inadequacy of text-space similarity metrics.
• Model-Aware Locality Metric: Introduction of Score-Prediction Distance as a superior metric for measuring concept proximity, replacing text-embedding similarities (like CLIP) with direct measurements of the model's denoising behavior.
• Dual-Component Framework: The integration of Locality-Aware Target Selection and Locality-Aware Replay, coupled via the same metric, to simultaneously minimize update magnitude and protect vulnerable neighbors.
• Stable 10-Step Unlearning: Demonstration that LACU maintains stable performance over 10 sequential unlearning steps, a regime where most baselines fail.

4. Experimental Results

The authors evaluated LACU on Stable Diffusion v1.5, sequentially unlearning 10 diverse concepts (objects, artistic styles, and individuals) across 5 random orderings.

• Performance vs. Baselines:
  • Retention: LACU significantly outperforms recent baselines (e.g., ESD, UCE, AGE, DUGE) in Related Retention Accuracy (RRacc) and General Retention Accuracy (GRacc). While aggressive baselines achieve high unlearning accuracy but suffer retention collapse (RRacc/GRacc dropping near 0 after 5–10 steps), LACU maintains high retention scores (e.g., RRacc $\approx$ 0.72, GRacc $\approx$ 0.87 after 10 steps).
  • Unlearning Efficacy: LACU achieves high Unlearning Accuracy (Uacc $\approx$ 0.90–0.99), effectively removing target concepts without sacrificing the ability to generate unrelated content.
• Ablation Studies:
  • Metric Choice: Using score-prediction distance for selection yields superior long-term stability compared to CLIP-based or naive fixed-anchor selection.
  • Component Contribution: Removing the local replay component leads to retention collapse due to compounding ripple effects. Removing regularization leads to parameter drift. The combination of both is necessary for the observed balance.
• Qualitative Analysis: Visual comparisons show that while baselines degrade the quality of retained concepts (e.g., failing to generate "Van Gogh style" after unlearning "Van Gogh"), LACU preserves the generative quality of retained concepts while successfully erasing the target.

5. Significance and Claims

The paper claims that locality-aware design is the key to stability in sequential unlearning. By treating unlearning as a series of minimal, targeted perturbations in the model's own representation space—rather than a global redirection—the framework prevents the accumulation of errors that leads to model collapse.

The authors emphasize that their approach does not require full retraining and is compatible with existing diffusion architectures. However, they note limitations:

• The method currently relies on an LLM to generate candidate prompts, though the core selection logic is model-agnostic.
• Style concepts remain challenging to unlearn due to broadly shared features.
• Real-world deployment may require scaling to hundreds or thousands of sequential deletions, which remains an open challenge.
• The teacher-student framework requires maintaining a frozen copy of the model, increasing memory and computational overhead compared to single-model approaches.

The code is made available at https://github.com/SonyResearch/LACU.