Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models

1. Problem Statement

Text-to-Image (T2I) diffusion models, trained on massive uncurated datasets, pose risks such as generating harmful content, reproducing copyrighted works, or mimicking specific artistic styles. Concept Erasure aims to remove these specific visual concepts from the model's generative capacity without retraining.

While Localized Concept Erasure (e.g., GLoCE) attempts to restrict edits to the spatial region where a concept appears to preserve the rest of the image, the authors identify a critical failure mode: the "Neighbor Gap."

• The Issue: Suppressing a target concept (e.g., a specific dog breed) often unintentionally weakens or degrades the generation of semantically adjacent concepts (e.g., other similar dog breeds).
• The Limitation: Existing training-free localized methods lack mechanisms to explicitly distinguish between the target concept and its semantic neighbors, leading to "collateral damage" in fine-grained domains.

2. Methodology: Neighbor-Aware Localized Concept Erasure (NLCE)

NLCE is a training-free, three-stage pipeline designed to erase target concepts while actively preserving the semantic structure of neighboring concepts. It operates entirely during inference.

Stage 1: Representation-Space Modulation

This stage operates at the token level to suppress the target subspace while reinforcing neighbor semantics.

• Target Subspace Identification: The authors use Singular Value Decomposition (SVD) on the embeddings of the target concept tokens ($F_c$) to construct a low-rank orthonormal basis ($U_{F_c}$).
• Spectral Weighting: A spectral expansion mechanism modulates the singular vectors based on their importance, creating a projection operator ($P_{F_c}$) that attenuates the target direction.
• Neighbor Mining & Reinjection:
  • Neighbors ($N_c$) are retrieved from an external pool (e.g., Wikipedia) using cosine similarity, filtered by concreteness and popularity, and re-ranked by visual CLIP similarity.
  • A neighbor projection operator ($P_{N_c}$) is constructed similarly to the target.
  • Final Operator: The method combines these to form a modulation operator: $P_c = (I - \beta P_{F_c}) + \gamma P_{N_c}P_{F_c}$. This suppresses the target while re-injecting neighbor information into the latent space.
• Implementation: This operator is applied globally to the Key and Value projection matrices in the UNet's cross-attention layers.

Stage 2: Attention-Guided Spatial Gating

Even after representation modulation, residual concept activations may persist in the attention flow. This stage localizes and suppresses them.

• Two-Pass Mechanism:
  1. First Pass (Dry Run): The model runs a forward pass with the modified embeddings. Attention maps are extracted from the UNet's DownBlock-2.
  2. Residual Detection: Tokens with high activation in the erased subspace are identified as "live tokens."
  3. Gate Construction: A spatial gate map ($G_t$) is generated by summing the attention maps of these live tokens. This highlights pixels where the erased concept still "leaks."
  4. Second Pass (Suppression): The model runs a second pass. The attention weights for live tokens are suppressed only in the spatial regions identified by the gate map ($G_t$), leaving unaffected regions intact.

Stage 3: Gated Feature Clean-up

To ensure irreversible removal of any remaining traces, a hard erasure is applied.

• Mask Generation: The spatial gate from Stage 2 is upsampled and thresholded to create a binary mask.
• Hard Erasure: In the final UNet layers, latent features within the masked regions are set to zero ($h_t \leftarrow \mathbf{0}$). This ensures that any residual target signals are physically removed from the generation process without affecting non-target areas.

3. Key Contributions

1. Identification of the Neighbor Gap: The paper formally defines and demonstrates that localized erasure methods often degrade semantically related concepts, a problem previously overlooked in training-free approaches.
2. NLCE Framework: A novel, training-free pipeline that integrates spectral representation modulation, attention-guided spatial gating, and hard feature clean-up. It is the first to explicitly model and preserve the "neighborhood" of a concept during erasure.
3. Modular Multi-Concept Support: The framework naturally extends to erasing multiple concepts simultaneously by composing projection operators, avoiding cross-concept interference.

4. Experimental Results

The authors evaluated NLCE on four distinct benchmarks:

• Fine-Grained Datasets (Oxford Flowers, Stanford Dogs):
  • Metric: Target Accuracy ($A_{ct}$), Retain Accuracy ($A_{cr}$), and Harmonic Mean ($H_o$).
  • Result: NLCE achieved the highest $H_o$ and $A_{cr}$ among all baselines (including MACE, GLoCE, RECE). It successfully erased specific dog breeds/flowers while maintaining high generation quality for visually similar breeds/flowers, whereas competitors like GLoCE showed significant neighbor degradation.
• Celebrity Identity Erasure:
  • Result: NLCE outperformed baselines in removing a target celebrity from a multi-person image while preserving the identity of the non-target person. It achieved lower LPIPS (perceptual difference) in non-target regions compared to GLoCE and SLD.
• Explicit Content (I2P Benchmark):
  • Result: NLCE significantly reduced the detection of nudity/explicit content (measured by NudeNet) while maintaining high CLIP scores (text-image alignment) for safe prompts, outperforming methods that either failed to erase or degraded general image quality.
• Artistic Style Erasure:
  • Result: NLCE effectively removed styles like "Van Gogh" or "Kelly McKernan" while preserving the ability to generate works in other styles (e.g., "Picasso"), demonstrating robust generalization.
• Efficiency:
  • NLCE is training-free. While it incurs a slight overhead due to neighbor retrieval and a two-pass UNet execution (~530 seconds for 10 concepts), it is significantly faster than training-based methods (e.g., SPM taking ~6951 seconds).

5. Significance

• Safety and Compliance: NLCE provides a practical, efficient solution for deploying T2I models in regulated environments where specific concepts (copyrighted artists, explicit content, private identities) must be removed without retraining the entire model.
• Semantic Precision: By addressing the "Neighbor Gap," NLCE advances the state of the art in fine-grained control. It proves that erasure does not have to be a blunt instrument that destroys semantic nuance; it can be precise enough to remove a specific entity while preserving its semantic family.
• Scalability: As a training-free method, it offers a scalable alternative to fine-tuning, making it accessible for rapid deployment and adaptation to new erasure requirements.

In conclusion, NLCE bridges the gap between global concept suppression and localized editing, offering a robust framework that balances erasure completeness with semantic fidelity to neighboring concepts.