Technical Summary: TILDE (TILt-based Distributional Erasure for Concept Unlearning)

Problem Statement

Concept unlearning in text-to-image diffusion models is essential for addressing privacy, copyright, and safety concerns, requiring models to suppress specific unwanted concepts (e.g., specific artists, characters, or objects) after training. However, existing methods face a critical trade-off: effective forgetting often leads to "retention failure," where the model degrades in generating benign, semantically related, or general prompts.

Current approaches typically rely on:

• Direct suppression: Penalizing concept scores, which often damages adjacent benign concepts (collateral damage).
• Anchor-based editing: Mapping the forget concept to a target concept, which introduces instability and bias, especially under sequential deletion requests.
• Reward-based optimization: Maximizing a scalar forgetting reward, which often collapses diversity by concentrating generation on a narrow set of safe outputs.

The fundamental issue is that these methods do not explicitly define the desired post-unlearning distribution. Consequently, retention is treated as an implicit consequence of the update rule rather than a controlled objective, leading to suboptimal distributional fidelity compared to a hypothetical "retain-only" model trained from scratch without the unwanted data.

Methodology: TILDE

The authors propose TILDE (TILt-based Distributional Erasure), which reframes concept unlearning as a constrained distributional alignment problem. Instead of local suppression or anchor replacement, TILDE seeks the conditional distribution that minimizes deviation from the pretrained model while satisfying a forgetting constraint.

1. Theoretical Formulation

The target is defined as the solution to a constrained Kullback-Leibler (KL) projection:
$$p^*_C = \arg \min_{p} \mathbb{E}_{y \sim \pi} [KL(p(\cdot | y) \parallel p_{pre}(\cdot | y))]$$
Subject to:
$$\mathbb{E}_{y \sim \pi, x \sim p(\cdot|y)} [E_C(x)] \le \delta_C$$

Where:

• $p_{pre}$ is the pretrained conditional distribution.
• $E_C(x)$ is a "forget energy" quantifying the evidence of the concept $C$ in image $x$.
• The solution takes the form of an energy-tilted Gibbs distribution:
  $$p^*_C(x | y) \propto p_{pre}(x | y) \exp(-\beta E_C(x))$$

This formulation ensures that concept-expressing images are down-weighted, while benign images preserve their relative pretrained likelihoods (anchor-free).

2. Practical Surrogate: Thresholded CLIP Energy

To instantiate the abstract energy $E_C$, TILDE uses a thresholded CLIP-based surrogate:

• Concept Probes: A set of text descriptors $Q_C$ (e.g., "Pikachu," "a photo of Pikachu") is used to compute the average CLIP similarity score $s_C(x)$.
• Thresholding: A forget energy $eE_C(x)$ is defined such that it is zero if $s_C(x) \le \tau$ (benign region) and increases exponentially if $s_C(x) > \tau$.
• Significance: The threshold $\tau$ creates a "no-penalty" region, preventing gradients from affecting benign images with weak concept similarity, thereby preserving local retention.

3. Optimization via Residual $\nabla$-GFlowNet

To realize this tilted distribution in a diffusion model, TILDE employs residual $\nabla$-GFlowNet training:

• Goal: Learn the score correction required to transform the pretrained sampler into the energy-tilted target.
• Mechanism: Instead of training a full sampler from scratch, the model learns a residual continuation score $h_\phi$ that adjusts the denoising trajectory.
• Loss Function: The training minimizes a forward and reverse loss ($L_{fwd}, L_{rev}$) to enforce local consistency between the adapted denoiser and the pretrained denoiser, guided by the terminal forget energy.
• Implementation: Updates are restricted to LoRA adapters on attention layers, ensuring parameter efficiency and a minimal update bias.

Key Contributions

1. Distributional Formulation: The paper formulates concept unlearning as a constrained distributional alignment problem, deriving an explicit, anchor-free, minimal-deviation Gibbs target.
2. TILDE Algorithm: It introduces a practical instantiation using thresholded CLIP energy and residual $\nabla$-GFlowNet training in the diffusion latent space.
3. Empirical Validation: The method is evaluated across objects, artistic styles, and characters, demonstrating improvements in forgetting effectiveness while maintaining higher retention and distributional fidelity compared to prior baselines.

Experimental Results

The authors evaluated TILDE on Stable Diffusion v1.5 across ~60 concepts (objects, characters, styles, nudity).

• Performance: TILDE achieves near-complete forgetting (e.g., 0.99 Unlearning Accuracy for characters) while significantly outperforming baselines in Related Retention (preserving semantically adjacent concepts) and General Retention.
• Comparison:
  • Hard Erasure (e.g., MACE): Achieves high forgetting but destroys retention (Related Retention drops to ~0.17).
  • Soft Suppression (e.g., ESD): Preserves retention but often fails to achieve sufficient forgetting.
  • Anchor-Based (e.g., CA, UCE): Improves quality but relies on explicit retain sets and still leaves residual concept evidence.
  • TILDE: Places on the Pareto frontier, achieving high forgetting without the need for explicit retain supervision or anchor concepts.
• Ablation Studies:
  • Removing the $\nabla$-form (using absolute flow targets) collapses forgetting performance.
  • Removing the residual component forces the full score field to absorb the update, degrading general retention.
  • Removing the energy threshold causes catastrophic retention collapse, confirming the necessity of the "no-penalty" region.
• Distributional Metrics: TILDE shows lower FADE (Functional Alignment Distance) and FID scores compared to baselines, indicating closer alignment to the ideal "retain-only" distribution.

Significance and Claims

The paper claims that TILDE addresses the "retention failure" inherent in current unlearning methods by explicitly defining the post-unlearning distribution. By framing unlearning as a minimum-deviation distributional alignment, TILDE avoids the pitfalls of:

• Collateral damage caused by direct suppression.
• Diversity collapse caused by reward maximization.
• Instability caused by fixed anchor mappings.

The authors argue that TILDE represents the "natural minimal modification" of a pretrained model that satisfies forgetting constraints. The use of thresholded energy ensures that benign concepts are not penalized, while the residual $\nabla$-GFlowNet framework provides an efficient mechanism to realize this distributional target in high-dimensional latent spaces. The work suggests that future unlearning research should prioritize defining the target distribution before selecting update rules.