💻 computer science

GuardPaint:SpeculativeSafetyDecodingforText-to-ImageGeneration

GuardPaint introduces a speculative decoding framework that enhances the safety of text-to-image diffusion models by employing a lightweight auditor to detect and surgically repair unsafe regions within the generation trajectory, effectively mitigating adversarial attacks while preserving image quality and prompt fidelity without modifying the base model.

Original authors: Shreyash Dhoot, Paras Dhiman, Arsh Abbas Naqvi, Aranbi Dutta, Aman Chadha, Vinija Jain, Amitava Das

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shreyash Dhoot, Paras Dhiman, Arsh Abbas Naqvi, Aranbi Dutta, Aman Chadha, Vinija Jain, Amitava Das

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern landscape of artificial intelligence, a class of systems known as text-to-image models has learned to translate written descriptions into vivid photographs. These tools operate by starting with a field of visual static and gradually refining it, step by step, until a clear picture emerges that matches the user's words. While this technology offers remarkable creative power, its ability to follow instructions precisely creates a significant vulnerability. If a user provides a carefully crafted, deceptive prompt designed to trick the system, the model can be steered away from safety guidelines and forced to generate images containing explicit nudity or graphic violence. Current safety measures often act like a gatekeeper at the entrance, blocking harmful requests before the image is made, or like a security guard at the exit, flagging bad images after they are finished. However, these methods leave the actual process of image creation unguarded, often resulting in a simple refusal to generate anything rather than a safe, corrected image.

Researchers have now introduced a new approach called GuardPaint, which acts as a safety mechanism inside the creation process itself. Instead of blocking the request or rejecting the final picture, this system monitors the image as it is being formed. At specific moments during the generation, a lightweight auditor scans the developing image to see if any unsafe elements are beginning to appear. If the auditor spots a problem, such as a region that looks like it is forming into something prohibited, it does not stop the whole process. Instead, it isolates just that small, troubled area and triggers a repair. A specialized tool then generates several safe alternatives for that specific spot, and a decision-making system chooses the best one. This chosen repair is carefully blended back into the main image, replacing the unsafe part with something compliant while leaving the rest of the picture untouched. The result is a system that can turn a potentially harmful generation into a safe, coherent image without needing to retrain the underlying model or change its core weights.

The effectiveness of this method was tested against a variety of sophisticated attacks designed to bypass existing safety filters. These attacks included techniques that used hidden meanings, swapped words for similar-sounding ones, or rewrote prompts to sound innocent while carrying harmful intent. The researchers applied GuardPaint to several different types of image generation models, including older designs and newer, more advanced ones. In every case, the system significantly reduced the success rate of these attacks. For instance, on one popular model, the rate at which harmful images were successfully generated dropped from over eight percent to less than two percent when the defense was active. Crucially, this safety improvement did not come at the cost of image quality or the accuracy of the original request. The repaired images remained visually sharp and true to the user's description, with the safety edits confined strictly to the problematic regions.

The process works by treating the image generation as a journey that can be paused and corrected. When the system detects that an image is veering off course, it does not simply delete the work. It acts more like a surgeon performing a precise operation, removing only the diseased tissue and grafting in healthy tissue that fits perfectly with the surrounding area. The researchers found that the timing of these checks matters; intervening too early, when the image is still mostly noise, is inefficient, while waiting too long allows the harmful structure to become too solid to fix easily. By intervening at the right moment, usually when the image is about eighty percent complete, the system can catch the issue while it is still manageable. The repair tool is trained to understand what a safe version of the image should look like, learning to replace explicit content with clothing or other appropriate elements that maintain the scene's context.

A key innovation in this work is the method used to select the best repair. The system does not just pick the first safe option it finds. Instead, it generates a small group of candidate repairs and evaluates them against strict criteria. Each candidate is scored on how well it removes the danger, how well it preserves the original meaning of the prompt, and how seamlessly it blends into the rest of the image. Only if a candidate scores high enough on all these fronts is it accepted. If none of the options are good enough, the system leaves the original image alone rather than risking a worse outcome. This ensures that the safety intervention never degrades the quality of a benign image or introduces new visual errors. The researchers also noted that while the system is highly effective, it is not perfect; it is trained to recognize specific categories of harm like nudity and violence, but it may not catch more subtle forms of bias or cultural insensitivity that do not manifest as explicit visual content.

The study demonstrates that safety in artificial intelligence does not have to be a blunt instrument that blocks creativity or refuses requests. By integrating a safety layer that operates during the generation process, it is possible to correct harmful outputs in real-time, offering a compliant alternative rather than a blank refusal. This approach works across different generations of image models without requiring the massive computational cost of retraining them from scratch. While the system adds a small amount of time to the generation process, the trade-off is a significant reduction in the risk of producing harmful content. The work suggests that the future of safe image generation lies in these dynamic, internal corrections, allowing these powerful tools to be used more broadly and responsibly without sacrificing their ability to create.

Technical Summary: GuardPaint

Problem Statement

Text-to-image (T2I) diffusion models possess high controllability, which inadvertently creates a critical safety vulnerability: adversarial prompts can steer the denoising trajectory toward policy-violating content, such as explicit nudity or graphic violence. Existing safety mechanisms primarily operate at the boundaries of the generation process. Prompt-level safeguards filter or rewrite inputs before sampling, while post-hoc classifiers intervene only after an image is fully generated. These methods often result in a binary "refusal" (blocking the generation entirely) rather than providing a safe visual alternative. Furthermore, methods that modify the generator itself (e.g., concept erasure, model editing, or preference alignment) often require extensive retraining, are architecture-specific, or fail to explicitly repair unsafe spatial regions during the decoding process.

The central question addressed by this work is: Can safety alignment be introduced as a modular decoding-time layer without modifying the underlying generator?

Methodology: GuardPaint

GuardPaint is a speculative safety decoding framework inspired by speculative decoding in large language models. It intervenes inside the diffusion trajectory using a "draft-audit-repair-verify" loop without altering the weights of the base T2I model. The system comprises three core components:

1. Auditor-Scorer (Detection and Localization)

A lightweight, multi-task model ( $A_s$ ) monitors intermediate images during the denoising process.

Architecture: Based on a ResNet-101 backbone with a BiLSTM text encoder for prompt conditioning and a timestep MLP for temporal awareness. It utilizes Feature-wise Linear Modulation (FiLM) to calibrate risk assessment differently for early noisy views versus late semantically formed images.
Outputs:
- Triggering: A binary probability indicating if intervention is needed.
- Localization: Per-class risk maps to identify unsafe spatial regions.
- Scoring: Metrics for prompt faithfulness, perceptual quality (seamlessness), and adversarial suppression strength.
Training: Trained on 82,000 image-prompt pairs with a factorized multi-task objective separating repair triggering, harm typing, and fidelity preservation.

2. Policy-Aligned Inpainter (Repair Generation)

A specialized inpainting model ( $A_g$ ) generates candidate repairs for localized unsafe regions.

Training Strategy: Follows a two-stage approach mirroring Supervised Fine-Tuning (SFT) followed by alignment in LLMs.
- Stage 1 (Refusal SFT): Fine-tuned on Stable Diffusion 1.5 Inpainting using LoRA. It learns to replace masked unsafe regions with safe, coherent content (e.g., clothing) conditioned on the original adversarial prompt.
- Stage 2 (Binary Classifier Optimization - BCO): Aligns the model to prefer safe completions over unsafe ones. Unlike standard preference optimization, BCO is used here to handle the asymmetric risk profile (false negatives are costlier than false positives). It optimizes a masked reconstruction reward, penalizing the model if it fails to improve upon a reference in safe regions or if it fails to suppress unsafe content in risky regions.
Key Innovation: The use of BCO with a "quadratic hinge" identity loss prevents the model from simply smudging pixels to minimize loss, forcing it to generate structurally coherent safe alternatives.

3. Guarded Tournament (Selection and Verification)

Instead of applying repairs directly, GuardPaint employs a tournament policy ( $\pi_\theta$ ) to select the best candidate.

Process: For a flagged region, the policy samples $N$ configurations (varying CFG scale, mask dilation, noise jitter, etc.). The inpainter generates $N$ candidates.
Scoring: The Auditor-Scorer evaluates each candidate. A candidate is accepted only if it passes a Guarded Utility function:
$u_i = (S_i - S_0 - \delta)_+ \cdot \mathbb{1}[P_i \geq \tau_P] \cdot \mathbb{1}[F_i \geq \tau_F] \cdot B_i$
Where $S$ is safety gain, $P$ is perceptual quality, $F$ is prompt faithfulness, and $B$ is suppression confidence.
Non-Regression Guard: If no candidate improves the safety objective while maintaining quality and fidelity, the original trajectory is preserved. This ensures the system never degrades benign generations.

4. Latent Reinsertion

To integrate the repair back into the active generation trajectory without disrupting the denoising manifold:

UNet-based models (SD 1.5, SDXL): Uses Null-text inversion to optimize a per-step null-text embedding, ensuring the edited latent aligns with the base model's trajectory.
Flow-matching models (SD 3.5, FLUX.1): Uses Flow-ODE reinsertion, projecting the edited clean latent forward to the current normalized time step.

Key Contributions

Plug-and-Play Safety Alignment: Introduces a decoding-time intervention layer that works across diverse T2I architectures (UNet and flow-matching) without retraining the base generator.
Speculative Safety Decoding: Proposes a framework combining adversarial auditing, localized inpainting, and guarded candidate ranking to convert unsafe generations into safe, semantically coherent visual alternatives rather than blank refusals.
Guarded Tournament Mechanism: Develops a selection mechanism that accepts edits only when they strictly improve policy compliance while preserving prompt fidelity and perceptual quality, preventing "safety at the cost of quality."
Empirical Validation: Demonstrates effectiveness across five jailbreak families (SneakPrompt, MMA, PGJ, DACA, RABell) and four generations of models (SD 1.5, SDXL, SD 3.5, FLUX.1-dev).

Results

Evaluated under the JailBreakDiffBench protocol, GuardPaint demonstrates significant reductions in Attack Success Rates (ASR) across all tested models and attack families:

UNet Models: On SD 1.5, ASR for DACA dropped from 8.25% to 1.94%, and SneakPrompt from 7.00% to 1.79%. On SDXL, MMA was reduced from 5.25% to 3.92%.
Flow-Matching Models: Despite starting from lower baselines due to stronger pretraining, GuardPaint achieved consistent reductions (e.g., FLUX.1-dev SneakPrompt reduced from 8.25% to 6.99%).
Quality and Fidelity: Prompt fidelity (AlignS) and image quality (BLIP) remained stable, staying within $\pm 0.05$ of the undefended baseline, confirming that repairs are localized and do not degrade the overall image.
Ablation: The system achieves diminishing returns beyond 5 candidates per tournament ( $N=5$ ) and auditing the final two denoising steps, balancing security with computational cost.

Significance and Limitations

Significance:
The paper argues that safety alignment should move from boundary-level filtering to trajectory-level repair. GuardPaint provides a modular, plug-and-play solution that offers "compliant alternatives" to adversarial outputs, addressing the limitation of existing methods that only offer refusal. It successfully bridges the gap between safety and usability by preserving the semantic intent of the prompt while surgically removing policy violations.

Limitations (as claimed by the authors):

Latency: The system introduces overhead (approx. 7–8 seconds per audited step for UNet models, up to 20 seconds for flow-matching models), though this is amortized by auditing only the final steps.
Single-Family Inpainter: The repair model is based on SD 1.5 Inpainting regardless of the base model. While reinsertion bridges the gap, a native inpainter for each architecture (e.g., FLUX.1) would better preserve stylistic properties.
Coarse Policy Taxonomy: The system is trained on three categories: {safe, nudity, violence}. It cannot detect subtler violations like bias, stereotyping, or cultural insensitivity that do not manifest as localized explicit content.
Adversarial Robustness: The system assumes the Auditor is reliable. A white-box adversary with knowledge of the Auditor's architecture could potentially craft inputs that suppress the Auditor's score while satisfying the base model.
Domain Gap: Performance may degrade on out-of-distribution styles (e.g., highly stylized art) where the Auditor's heatmap generation is less accurate, potentially leading to incorrect repairs or failure to trigger.
Static Calibration: Thresholds are calibrated offline on a fixed distribution of attacks; periodic recalibration is necessary as jailbreak methods evolve.