Technical Summary: SACE: Concept Erasure at the Semantic Singularity in Visual Autoregressive Models

1. Problem Statement

The rapid emergence of Visual Autoregressive (VAR) models, such as Infinity, has established a new frontier for high-fidelity text-to-image synthesis, surpassing diffusion models in scalability and generation quality. However, these models inherit significant safety vulnerabilities, including the generation of inappropriate, biased, or copyrighted content.

Existing concept erasure techniques, predominantly designed for diffusion models, fail when naively applied to VAR architectures. Diffusion erasure relies on the homogeneous nature of temporal denoising steps, whereas VAR models operate via a "next-scale" autoregressive paradigm where images are generated from coarse to fine resolutions. Directly applying uniform erasure losses across all scales in VAR models leads to:

• Catastrophic Semantic Collapse: The model loses the ability to generate coherent images.
• Visual Artifacts: Severe blurriness, chaotic textures, and structural disintegration.
• Inefficiency: Indiscriminate training across all scales incurs exponentially growing computational overhead without guaranteeing effective erasure.

Current VAR-specific attempts (e.g., S-VARE, EAR) exhibit sub-optimal efficacy and high training costs, largely because they treat the multi-scale generation process as a homogeneous sequential task, failing to grasp the intrinsic spatial-hierarchical nature of these models.

2. Methodology

The proposed SACE (Scale-Aware Concept Erasure) framework addresses these challenges through a three-pronged approach: theoretical axiomatization, diagnostic analysis, and targeted optimization.

A. The Semantic Singularity Axiom

The core theoretical contribution is the Semantic Singularity Axiom, which posits that any target semantic concept embedded in a prompt is irrevocably locked at the foundational generative scale (Scale-0).

• Mechanism: The global text condition initiates a "visual accumulator cascade." The semantic commitment point is established at Scale-0 (a $1 \times 1$ latent token) and propagates structurally to all higher-frequency scales.
• Implication: Subsequent scales ($s > 0$) do not inject new semantic concepts; they merely perform high-frequency structural locking and texture refinement based on the priors established at Scale-0. Therefore, erasure interventions across all scales are redundant and structurally detrimental.

B. Incremental Semantic Saliency Analysis (ISSA)

To validate the axiom, the authors propose ISSA, a diagnostic framework that isolates newly injected semantic information at each resolution scale.

• Process: ISSA intercepts cross-attention probabilities during inference, aggregates them across layers, and computes the Attention Residual ($\Delta A_k$) by subtracting the upsampled attention of the previous scale from the current scale.
• Findings: Visual analysis reveals that attention mass for specific concepts peaks at Scale-0 and experiences an exponential decay in subsequent scales. Higher scales contribute negligible new semantic focus, acting solely as texture refiners. This provides rigorous visual proof that intervening at scales $s > 0$ is both unnecessary and harmful.

C. The SACE Optimization Framework

Guided by the axiom, SACE confines all erasure interventions strictly to Scale-0, employing a dual-objective optimization strategy:

1. Entropy-Regularized Erasure Objective ($\mathcal{L}_{ER}$):

   • Problem: Forcibly displacing latent representations in autoregressive models flattens output logits, leading to pathologically high entropy and near-random token selection (degeneration).
   • Solution: $\mathcal{L}_{ER}$ couples a generative deflation mechanism (repelling the target concept) with a dynamic information-theoretic sharpness constraint. It ensures the model erases the concept without collapsing the categorical distribution into high-entropy noise.
   • Formula: $\mathcal{L}_{ER} = \mathcal{L}_{naive} + \lambda_{ent}\mathcal{L}_{ent}$, where $\mathcal{L}_{ent}$ penalizes entropy exceeding a maximum threshold ($H_{max}$).

2. Restorative Preservation Loss ($\mathcal{L}_{pre}$):

   • Problem: Ablating the target concept can damage fine-grained cross-token dependencies necessary for high-fidelity synthesis, causing "collateral damage" to benign priors (e.g., distorted anatomy).
   • Solution: $\mathcal{L}_{pre}$ anchors the integrity of entangled benign priors by minimizing the distance between the erased model's output and the original model's output on non-target prompts, specifically on scales $s \ge 1$.
   • Role: This acts as a "surgical scalpel," forcing the model to perform orthogonal disentanglement at Scale-0 while preserving the high-frequency structural gradients of higher scales.

3. Key Contributions

1. Semantic Singularity Axiom: The first formalization that semantic injection in next-scale autoregressive models is dominated by the initial scale, simplifying multi-scale erasure into a targeted intervention at the semantic root.
2. Incremental Semantic Saliency Analysis (ISSA): A white-box diagnostic tool that isolates incremental semantic information, providing the community with the first method to transparently inspect the coarse-to-fine generation process of VAR models.
3. Scale-Aware Erasure Framework: The first framework to migrate mature diffusion-based erasure strategies to the VAR paradigm by leveraging the Semantic Singularity Axiom. It combines First-Scale Intervention with bi-level optimization.
4. Surgical Precision: The method achieves state-of-the-art (SOTA) erasure performance with minimal training overhead, effectively balancing concept removal with the preservation of general generative fidelity.

4. Experimental Results

The authors evaluated SACE on the Infinity-8B model against baselines including ESD, FMN, UCE, and VARE (reproduced).

• NSFW Erasure ("Nude"):

  • SACE reduced the Human-judged Nudity Rate (HNR) from 97.7% (baseline) to 17.5%.
  • Unlike aggressive baselines (e.g., ESD-x) that achieved low HNR (7.2%) but destroyed image quality (FID 52.18), SACE maintained a competitive FID of 23.5 and CLIP score of 31.3, nearly indistinguishable from the original model.
  • SACE achieved the lowest sensitivity score (71) on anatomical detections while maintaining high complexity (375), indicating preserved generative richness.

• Style Erasure ("Van Gogh"):

  • SACE reduced style presence to 8% accuracy, outperforming baselines (18–28%).
  • Crucially, SACE maintained an FID of 23.8, whereas ESD-u suffered catastrophic utility degradation (FID 88.1).

• Object Erasure ("Church"):

  • SACE was the only method to significantly disrupt the generation of concrete objects, reducing accuracy from 97.0% to 65.0%. Baselines failed to provide meaningful erasure (88–98%).

• Robustness:

  • SACE demonstrated strong robustness against adversarial attacks (Ring-A-Bell and I2P datasets), achieving the lowest Attack Success Rate (ASR) on R-A-B (44.56%) and second-best on I2P.

• Ablation Studies:

  • Scale: Intervening on all scales degraded erasure efficacy (ASR rebounded to 55.79%) due to objective conflicts between erasure and preservation forces.
  • Entropy: Removing entropy regularization led to suboptimal erasure (ASR 54.04%) due to logit flattening.
  • Preservation: Without $\mathcal{L}_{pre}$, generative quality collapsed (FID 27.4). A strong anchor ($\lambda_{pre}=2.0$) was required to force precise disentanglement.

5. Significance and Claims

The paper claims to provide the first scale-aware concept erasure framework specifically designed for Visual Autoregressive models. Its significance lies in:

1. Foundational Insight: It fundamentally redefines how safety interventions should be approached for VAR models, moving away from the "one-size-fits-all" approach of diffusion models to a scale-specific strategy.
2. Efficiency and Safety: By restricting interventions to Scale-0, the method drastically reduces computational costs while mathematically guaranteeing that high-frequency structural details remain untouched.
3. Bridging Paradigms: It offers a systematic methodology to adapt diffusion-era safety research to the next generation of AR models, leveraging years of prior research to solve emerging safety vulnerabilities.
4. Surgical Precision: The framework achieves a delicate balance, removing sensitive concepts without the "collateral damage" of blurring or destroying the model's general generative capabilities, a critical requirement for real-world deployment.

The authors conclude that their approach "timely and elegantly resolves" the critical safety vulnerabilities inherent in emerging VAR architectures, providing a robust path toward the safe deployment of these high-fidelity generative models.