Technical Summary: Orthogonal Concept Erasure for Diffusion Models

Problem Statement

Text-to-image (T2I) diffusion models possess remarkable generative capabilities but are prone to producing undesirable, unsafe, or copyrighted content. While concept erasure has emerged as a critical solution to mitigate these risks, existing approaches face significant trade-offs between effectiveness, efficiency, and the preservation of general generative capacity.

Current methods generally fall into three categories:

1. Inference-time intervention: Adjusts sampling trajectories without modifying parameters. These are vulnerable to bypassing.
2. Training-based methods: Fine-tune model parameters to remove concepts. While effective, they incur high computational costs and require multiple optimization rounds, limiting scalability.
3. Editing-based methods: Modify model parameters directly using closed-form solutions. While efficient, existing editing methods typically rely on additive parameter updates ($W + \Delta$). The authors identify a core limitation in this paradigm: additive updates inherently entangle changes in neuron direction, magnitude, and angular geometry. Since concept semantics rely heavily on neuron direction while overall generative capacity depends on the preservation of angular geometry, additive updates often result in unstable erasure and the degradation of non-target concepts.

Methodology: Orthogonal Concept Erasure (OCE)

The paper proposes Orthogonal Concept Erasure (OCE), a geometry-driven, editing-based method that reformulates concept erasure as a multiplicative orthogonal transformation rather than an additive perturbation.

1. Geometric Analysis

The authors establish two key empirical claims through controlled experiments:

• Claim 1 (C1): Concept semantics in diffusion models are primarily encoded in neuron directions, while neuron magnitudes have negligible effects on semantic expression.
• Claim 2 (C2): Preserving the angular geometry (inter-neuron angles and correlations) is critical for maintaining the model's overall generative quality.

Based on this, OCE avoids additive updates which disturb both direction and magnitude. Instead, it applies a layer-wise orthogonal transformation $P$ to the projection matrices $W$ (specifically the key and value matrices in cross-attention layers), such that $W^* = PW$. This operation rotates neuron directions to erase target concepts while strictly preserving neuron magnitudes and inter-neuron angular geometry.

2. Problem Formulation

The method formulates erasure as an optimization problem to find an orthogonal matrix $P$ ($P^\top P = I$) that minimizes the distance between transformed target concepts and their anchors, while minimizing the distance between transformed non-target concepts and their original representations.

• Vector-wise Objective: For single-concept erasure, the objective aligns target embeddings $C_1$ with anchor embeddings $C^*$ and preserves retain set $C_0$. This is solved via the classical Orthogonal Procrustes problem using Singular Value Decomposition (SVD) to obtain a closed-form solution: $P = UV^\top$.

• Subspace-level Objective: For multi-concept erasure, enforcing strict vector-wise alignment for multiple targets creates conflicting constraints. OCE extends the objective to a subspace-level projection. It defines target and anchor subspaces and seeks to suppress the components of the target subspace that lie in the orthogonal complement of the anchor subspace. This is achieved by minimizing the energy of the target subspace outside the anchor's orthogonal complement, combined with vector-wise preservation for the retain set.

3. Implementation Details

• Global Preservation Prior ($K_0$): To ensure global stability, the method precomputes a generic preservation prior $K_0$ from a large dataset (COCO-30k), which is reused across different erasure tasks.
• Asymmetric Design: The method employs an asymmetric formulation: subspace-level erasure (to handle conflicts in multi-concept scenarios) combined with vector-wise preservation (to ensure fine-grained retention of non-target concepts).
• Adversarial Robustness: The framework can be seamlessly integrated with adversarial editing strategies to improve robustness against attacks.

Key Contributions

1. Geometric Reformulation: OCE introduces a paradigm shift from additive to multiplicative orthogonal editing, theoretically grounded in the observation that concept semantics depend on direction while generative capacity depends on angular geometry.
2. Closed-Form Solution: The method provides a principled, efficient closed-form solution for both single- and multi-concept erasure via SVD, eliminating the need for iterative optimization.
3. Scalable Subspace Erasure: By extending vector-wise erasure to a structured subspace-level objective, OCE effectively resolves conflicting constraints in large-scale multi-concept erasure (e.g., erasing 100 concepts simultaneously).
4. Broad Applicability: The method demonstrates transferability to both U-Net-based models (Stable Diffusion v1.4) and DiT-based models (FLUX.1 dev).

Experimental Results

Extensive experiments were conducted on object erasure, artistic style erasure, celebrity erasure, and implicit concept erasure (including adversarial attacks).

• Single-Concept Erasure: On CIFAR-10 object erasure, OCE achieved the highest harmonic mean score ($H_o$), reducing the average erasure failure rate ($A_{cce}$) to 4.61% while maintaining near-perfect preservation of non-target concepts ($A_{ccs} \approx 98.68\%$). It outperformed state-of-the-art methods like UCE, MACE, and SPEED.
• Multi-Concept Erasure: OCE successfully erased up to 100 celebrity concepts in a single step, completing the process in 4.3 seconds on a single A100 GPU. It maintained stable performance as the number of erased concepts increased, whereas other methods suffered from generation collapse or significant degradation in non-target preservation.
• Implicit Erasure & Robustness: In implicit concept erasure (e.g., NSFW content) and under adversarial attacks (Ring-A-Bell, MMA-Diffusion), OCE demonstrated strong robustness, particularly when integrated with adversarial editing, achieving low Attack Success Rates (ASR) while preserving general generation quality (high CLIP scores, low FID).
• Transferability: The method effectively extended to the FLUX.1 dev model, achieving effective erasure of objects, styles, and celebrities without requiring architectural changes to the core editing logic.

Significance and Claims

The paper claims that OCE offers a principled, efficient, and scalable solution for concept erasure. By addressing the geometric limitations of additive updates, OCE achieves a superior balance between precise concept removal and the preservation of the model's intrinsic generative capabilities.

The authors emphasize that OCE provides a controllable mechanism to improve the safety and reliability of generative AI models. It supports both single- and multi-concept erasure with minimal interference on unrelated concepts, offering a practical pathway for deploying safer AI in content creation, education, and public platforms. The work suggests that geometric perspectives on parameter space can lead to more effective and stable editing paradigms for diffusion models.

Limitations Acknowledged:
The authors note that SVD-based subspace erasure may introduce computational overhead for very large models. Additionally, because the method operates at the subspace level, generated outputs may fall into intermediate semantic regions rather than precisely matching fine-grained anchor concepts, which could limit applicability in specific editing tasks. Further exploration is needed for erasing complex relational or compositional concepts.