SUGAR: A Sweeter Spot for Generative Unlearning of Many Identities

1. Problem Statement

Generative AI models (specifically 3D-aware models like EG3D) can inadvertently memorize and reproduce identifiable faces from their training data, posing significant risks to privacy and intellectual property. While the "right to be forgotten" allows users to request data removal, retraining models from scratch is computationally prohibitive.

Existing Generative Unlearning methods face three critical limitations when handling multiple identities:

1. Scalability: They struggle to handle simultaneous or sequential unlearning requests for many identities.
2. Leakage: Suppression mechanisms often map forgotten identities to "noise" or a static "average face," which can still leak information about the original identity or be detected via membership inference attacks.
3. Utility Degradation: Aggressive unlearning often causes "semantic drift," degrading the quality and diversity of identities that were not requested to be forgotten (retained identities).

2. Methodology: SUGAR Framework

SUGAR (Scalable Unlearning of Generative Identities via Adaptive Representations) is a framework designed to remove multiple identities without retraining the entire model. It operates within a pre-trained EG3D-style pipeline (Encoder $E_\psi$, Generator $G_\theta$, Renderer $R$) and updates only the generator.

The framework consists of two main phases:

A. ID-Specific De-Identification (Phase 1)

Instead of mapping all forgotten identities to a single static target (like a mean face) or noise, SUGAR learns a personalized surrogate latent for each identity.

• Mechanism: It trains a mapping function $\Phi$ (or $T_\Theta$) to transform the latent vector of a forgotten identity ($w_u$) into a new, counterfactual target latent ($w_t$).
• Objective: The new target $w_t$ must:
  1. Preserve coarse facial attributes (structural plausibility) to maintain visual coherence.
  2. Be distinct enough from the original identity to prevent recognition (measured by ID loss and perceptual loss).
• Mathematical Formulation: The target is defined as $w_t = \bar{w} - d \cdot \frac{T_\Theta(w_{id})}{\|T_\Theta(w_{id})\|}$, where $\bar{w}$ is the average latent, $w_{id}$ is the identity vector, and $d$ is a controllable displacement factor. This allows for controllable unlearning, where the degree of forgetting can be tuned.

B. Forgetting in a Continual Learning Fashion (Phase 2)

SUGAR optimizes the generator using a dual-objective loss function to balance forgetting and retention.

1. Forgetting Loss ($L_{forget}$):
   • Applies the remapping objective to the anchor latent ($w_u$) and its vicinity neighbors ($w_{u,a}$).
   • Sampling neighbors ensures that the entire local region of the forgotten identity is suppressed, preventing residual attributes from leaking.
2. Retention & Utility Preservation:
   • Vicinity Sampling: Identifies latent points near the forgetting set that are prone to collateral damage.
   • Elastic Weight Consolidation (EWC): Instead of adding explicit reconstruction losses for every retained sample (which is computationally expensive), SUGAR uses EWC. It calculates the Fisher Information Matrix (FIM) on the vicinity probe set to identify parameters critical for retaining those specific identities.
   • Regularization: A penalty term ($L_{ewc}$) constrains the drift of these critical parameters from their pre-unlearning values, preserving the model's utility for non-forgotten identities.

Final Objective:
$$L_{unlearn} = \mathbb{E}_{w_u \sim W_u}[L_{forget}(w_u; \theta)] + \lambda_{ewc} L_{ewc}(\theta; \theta^*)$$

3. Key Contributions

1. First Multi-Identity GUI Method: SUGAR is the first Generative Identity Unlearning (GUI) method capable of handling both simultaneous and sequential removal of many identities (up to 200 tested) without retraining.
2. Personalized Surrogate Latents: Unlike methods mapping to noise or a fixed average, SUGAR learns a unique, visually coherent alternative for each forgotten identity, preventing information leakage and maintaining diversity.
3. Continual Utility Preservation: The integration of EWC and vicinity sampling effectively guards against the degradation of retained identities, a common failure point in previous baselines.
4. Controllability: The displacement factor $d$ allows model vendors to tune the trade-off between the strictness of forgetting and the preservation of model utility.

4. Experimental Results

The authors evaluated SUGAR against the state-of-the-art baseline GUIDE across three scenarios: Random, In-Domain (FFHQ), and Out-of-Domain (CelebA-HQ).

• Retention Utility: SUGAR achieved a 700% improvement in retention utility compared to baselines.
  • ID Similarity: For retained identities, SUGAR maintained ID scores roughly 2x higher than GUIDE (e.g., 0.57 vs 0.33 for N=1 in FFHQ).
  • FID Score: SUGAR significantly reduced Fréchet Inception Distance (FID), indicating better image quality and distribution alignment for retained faces.
• Forgetting Effectiveness:
  • SUGAR successfully reduced the ID similarity of forgotten identities by an average of 54.7% compared to retained ones.
  • Human Judgment: In a study with 60 participants, SUGAR-generated retained identities were correctly identified 86% of the time, whereas GUIDE's were only 1.9%. Conversely, participants could not distinguish original forgotten identities from unlearned outputs for either method, proving effective forgetting.
• Sequential Unlearning: In multi-stage experiments, GUIDE suffered from cumulative collateral damage (retained faces degraded over time), while SUGAR maintained high fidelity for retained identities across all stages.
• Privacy & Security: SUGAR maps forgotten identities to diverse counterfactuals on the data manifold, making them indistinguishable from natural variations. This prevents "template-matching" and membership inference attacks that succeed against noise-based or average-face methods.

5. Significance

SUGAR addresses a critical gap in the ethical deployment of generative AI. By enabling the scalable, sequential, and secure removal of specific identities without compromising the model's overall performance, it provides a practical solution for the "right to be forgotten" in high-fidelity image synthesis. The method's ability to learn personalized surrogates rather than collapsing to noise ensures that the model remains useful and diverse, making it a robust framework for future privacy-preserving generative systems.