AEGIS: A Mechanism-Guided Defense against Visual Synonym Jailbreaks in Text-to-Image Models

Technical Summary: AEGIS – A Mechanism-Guided Defense against Visual Synonym Jailbreaks

1. Problem Statement

Text-to-image (T2I) diffusion models face a critical safety-utility dilemma. Existing defense paradigms—ranging from input-space semantic sanitization to structural feature pruning—are primarily designed to intercept explicit unsafe concepts or known adversarial tokens. However, these methods fail against Visual Synonym Attacks (VSA).

VSA is a stealthy jailbreak technique where adversaries use contextually benign, natural language prompts (e.g., "spilled red paint") that are semantically orthogonal to prohibited concepts in the text domain (e.g., "blood") but share strong implicit visual associations. While text-centric filters and trigger-specific pathway disruptions successfully block explicit malicious prompts, they are bypassed by VSA because the input tokens lack explicit harmful semantics. Conversely, broad structural pruning intended to block VSA often leads to over-mitigation, degrading the generation quality of visually similar benign concepts (e.g., suppressing "ketchup" while trying to block "blood").

The core challenge identified is that VSA exploits a text-visual semantic misalignment: the prompt is safe in text space, but the generation trajectory dynamically converges to an unsafe visual manifold during the denoising process.

2. Methodology: Mechanistic Analysis and AEGIS

The authors propose AEGIS (Adaptive Evasion Guard via Identification and Steering), a defense mechanism that shifts from static suppression to dynamic tracing of unsafe semantics during generation.

A. Mechanistic Analysis (The "Why")
Through a systematic investigation of latent trajectories, the authors identify that VSA and explicit attacks converge through a sparse subset of semantic-injecting attention heads.

• Dynamic Convergence: Visual synonyms initially resemble benign prompts but dynamically diverge and converge toward malicious signatures during intermediate denoising steps.
• Sparse Localization: Using anchor-based similarity profiling and sparse regression (Lasso), the authors isolate that approximately 10% of attention heads (specifically in deep cross-attention layers) carry nearly 70% of the unsafe semantic attribution. These heads serve as the critical bottlenecks where harmless text embeddings are transformed into prohibited visual semantics.
• Shared Pathways: These identified heads are activated by both implicit visual synonyms and explicit malicious triggers, confirming they are the shared conduits for unsafe visual generation.

B. AEGIS Defense Mechanism (The "How")
AEGIS operates as an inference-time intervention that surgically steers these identified vulnerable heads without retraining the model or permanently ablating weights.

• Repulsion Beacon: The system uses an anchor embedding (representing the target unsafe concept) as a "repulsion beacon" within each identified critical head.
• Similarity-Aware Adaptive Gating: Instead of static projection, AEGIS applies a dynamic repulsion force scaled by the activation magnitude and a similarity threshold.
  • If the activation is highly similar to the malicious anchor, the gate opens fully to enforce maximum repulsion.
  • If the activation is borderline or benign, the gate scales linearly to apply subtle corrections, preserving utility.
  • This mechanism ensures that high-risk semantics are neutralized while benign concepts (even those with visual similarities) are preserved.

3. Key Contributions

1. Formalization of VSA: The paper formally defines Visual Synonym Attacks as a distinct jailbreaking paradigm that exploits text-visual semantic misalignment, highlighting the blind spots in current alignment paradigms.
2. Mechanistic Interpretability: The authors provide the first mechanistic analysis revealing that unsafe visual semantics in T2I models are governed by a sparse set of deep semantic-injecting attention heads, which serve as shared pathways for both explicit and synonym-based attacks.
3. AEGIS Framework: They propose a novel inference-time defense that applies surgical, similarity-aware repulsion exclusively at these vulnerable heads. This approach avoids the over-mitigation associated with broad feature pruning.
4. Comprehensive Evaluation: The method is evaluated against 16 baselines across multiple attack types (VSA, explicit, adversarial) and architectures (SD 1.4, SD 2.1, FLUX.1).

4. Experimental Results

Evaluated on Stable Diffusion 1.4 (SD 1.4), AEGIS demonstrates state-of-the-art performance:

• Safety Alignment:
  • In-Domain (VSA): Reduces Attack Success Rate (ASR) to 0.00 for violence and 0.03 for nudity.
  • Out-of-Domain (Explicit/Adversarial): Achieves ASRs ≤0.09 across I2P, MMA, and RAB benchmarks.
  • It significantly outperforms 16 baselines, including input sanitizers (SafeClip, SAFREE) and pathway disruptors (ESD, UCE).
• Utility Preservation:
  • Maintains high generation fidelity (FID scores comparable to unaligned models) and semantic alignment (CLIP scores).
  • Crucially, it avoids the over-mitigation of "hard-negative" benign concepts (e.g., "chopping tomatoes" or "mannequins") that other methods erroneously suppress.
• Transferability: The method successfully transfers to SD 2.1 and the Diffusion Transformer-based FLUX.1 after re-identifying the critical heads for each backbone, demonstrating that the vulnerability is structural rather than model-specific.
• Efficiency: AEGIS introduces minimal computational overhead (3.99s per image on SD 1.4), significantly faster than other inference-time interventions like SAeUron (11.60s), and requires no retraining of the diffusion backbone.

5. Significance and Claims

The paper claims that AEGIS breaks the "inescapable dilemma" of current safety alignments, which must choose between under-mitigating stealthy attacks or over-mitigating benign utility. By shifting the defense paradigm from static text-level filtering to dynamic internal mechanism tracing, AEGIS provides a practical solution that:

• Neutralizes the Root Cause: It targets the specific attention pathways where unsafe semantics emerge, rather than relying on surface-level text patterns.
• Ensures Robustness: It remains effective even against white-box adaptive attacks where adversaries attempt to evade the specific intervention directions, as the defense captures robust, safety-relevant model components.
• Offers Deployability: As a plug-and-play inference-time monitor that does not require retraining, it offers a low-maintenance, high-reliability solution for securing T2I models against evolving jailbreak techniques.

The authors conclude that their work provides a reliable blueprint for developing resilient safety mechanisms, demonstrating that the vulnerabilities exploited by VSA are structural and can be addressed through precise, mechanism-guided intervention.