💻 computer science

FASDP: Frequency-Aware Scale-Diverse Purification against Pixel-Space Visual Jailbreaks

The paper proposes FASDP, a training-free, frequency-aware purification framework that adaptively allocates defense strength across image scales to effectively mitigate pixel-space visual jailbreak attacks in vision-language models while preserving their performance on benign tasks.

Original authors: Xiaolong Qin, Chunjiang Wu, YiLe Li, TingHui Chen, Zhibin Li, ShiLin Qiu

Published 2026-11-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaolong Qin, Chunjiang Wu, YiLe Li, TingHui Chen, Zhibin Li, ShiLin Qiu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a new class of systems has emerged that can see and speak simultaneously. These vision-language models act as digital observers, capable of describing a photograph, answering questions about a diagram, or holding a conversation while looking at an image. They have become powerful tools for everything from medical imaging to autonomous driving. However, just as a human can be tricked by a visual illusion, these systems can be manipulated. Researchers have discovered that by making tiny, often invisible changes to the pixels of an image, attackers can force these safety-aligned models to ignore their ethical guidelines and generate harmful or dangerous responses. This vulnerability, known as a visual jailbreak, poses a significant threat because it does not require hacking the software itself; it only requires sending a corrupted image. The challenge for defenders is to clean up these corrupted images without blurring the picture so much that the model can no longer understand what it is looking at.

A team of researchers at the Chengdu University of Information Technology and the Chengdu University of Technology has proposed a new method to solve this delicate balancing act. They call their approach FASDP, a system designed to act as a filter that removes the malicious noise from an image while preserving the original scene. Unlike previous methods that apply a fixed level of cleaning to every image, this new system adapts its strength based on the specific characteristics of the image it is examining. The core idea is that not all images are attacked in the same way, and not all images need the same amount of cleaning. By measuring how the high-frequency details of an image—such as fine textures and sharp edges—change when the image is slightly resized, the system can determine how much "noise" is likely present. This measurement serves as a guide, telling the system whether to apply a gentle touch or a more aggressive scrubbing.

The process works by creating several versions of the same image, each processed at a different scale or level of detail. Some versions are treated lightly to keep fine details, while others are smoothed out more heavily to remove potential attacks. The system then uses a special mathematical rule to blend these versions together. It favors the versions that remain most consistent with the low-frequency, broad shapes of the original image, while using the frequency measurement to decide how much weight to give to the heavily smoothed versions. If the system detects that the image has been significantly altered by an attack, it leans more heavily on the smoothed versions to remove the threat. If the image appears clean, it preserves the fine details. Crucially, this entire process happens without needing to retrain the artificial intelligence model or access its internal workings, making it a flexible defense that can be applied to any existing system.

When the researchers tested their method against three different types of visual attacks on three popular vision-language models, the results were compelling. The system successfully reduced the rate at which these models produced harmful responses, often outperforming existing defenses like standard image compression or more complex diffusion-based filters. Perhaps more importantly, the method did not ruin the quality of the images. In tests where the models were asked to describe clean, unattacked images, the system maintained their ability to answer questions and generate captions with high accuracy. This suggests that the filter is precise enough to target only the malicious alterations without blurring the legitimate content. The researchers found that the system works effectively across a wide range of attack intensities, from subtle manipulations to more aggressive pixel-level distortions.

The study also explored how the system behaves when faced with attackers who know about the defense and try to adapt their strategies. Even when the attackers attempted to bypass the filter by optimizing their attacks specifically against it, the system remained more robust than other methods. The researchers noted that while the defense is highly effective against pixel-based attacks, it is not a universal solution for every type of security risk. It cannot stop attacks that rely on harmful text prompts or visible content within the image itself, as these do not involve the same kind of pixel-level noise. However, for the specific problem of invisible pixel manipulation, the findings offer a strong, practical solution. By adapting the cleaning strength to the needs of each individual image, the researchers have demonstrated a way to keep these powerful visual AI systems safe without sacrificing their ability to see the world clearly.

Technical Summary: FASDP – Frequency-Aware Scale-Diverse Purification against Pixel-Space Visual Jailbreaks

1. Problem Statement

Vision-Language Models (VLMs) have achieved significant success in applications like image captioning and visual question answering (VQA). However, their deployment has exposed vulnerabilities to visual jailbreak attacks. Unlike traditional text-based jailbreaks, these attacks manipulate the input image in pixel space to induce safety-aligned models to generate harmful responses.

The specific challenge addressed by this paper is the defense against pixel-space adversarial perturbations (e.g., HADES, Image Hijacks, and visual adversarial examples). These attacks often involve large $\ell_\infty$ budgets, perturbing a significant fraction of pixels. Existing defense strategies face a critical trade-off:

Mild transformations (e.g., resizing, JPEG) often fail to suppress strong perturbations.
Aggressive purification (e.g., high-noise diffusion) effectively removes perturbations but introduces artifacts and degrades the performance of the VLM on benign, clean inputs.
Fixed-strength defenses cannot adapt to the varying characteristics of different images and attack intensities.

The goal is to develop a training-free, input purification method that suppresses high-budget jailbreak perturbations while minimizing damage to the utility of clean images, without requiring access to the target model's parameters, gradients, or embeddings.

2. Methodology: FASDP

The authors propose FASDP (Frequency-Aware Scale-Diverse Purification), a framework that adaptively allocates purification strength on a per-image basis. The method operates entirely on the input image before it reaches the target VLM.

Core Components

Self-Referenced Frequency Analysis:
- Instead of relying on target-model signals, FASDP uses a self-referenced frequency statistic ( $\Delta$ ) to gauge the sensitivity of an input to transformation.
- It calculates the normalized change in the Discrete Cosine Transform (DCT) high-frequency energy ratio under a mild reference transformation (e.g., resizing to 0.75 scale).
- $\Delta$ serves as a heuristic cue: a positive value indicates that the transformation reduces high-frequency energy, suggesting the presence of perturbations that can be attenuated. This statistic modulates the fusion weights but does not act as a binary attack detector.
Scale-Diverse Candidate Generation:
- The framework generates multiple candidate images by applying a fixed set of scale transformations ( $S = \{1.0, 0.75, 0.5\}$ ) to the attacked image.
- These candidates are processed by a single, frozen, shared denoiser (a U-shaped window-attention–convolution network, specifically SCUNet).
- Crucially, the denoising strength ( $\eta$ ) is not fixed. It is calibrated per image based on the frequency analysis of multiple reference scales. The strength $\eta$ controls the interpolation between the scale-transformed image and the full denoiser output: $D(Y; \eta) = (1-\eta)Y + \eta g_\Theta(Y)$ .
Low-Frequency-Consistency Fusion:
- The final purified image is a weighted sum of the denoised candidates.
- The weights are determined by a consistency scoring rule that balances two factors:
  - Input Consistency: How much a candidate alters the low-frequency content of the original attacked image.
  - Pairwise Consistency: How much a candidate disagrees with other scale candidates.
- These scores are modulated by the frequency statistic ( $\Delta$ ) and a fidelity bias ( $\beta$ ) that favors larger scales (milder transformations) to preserve detail.
- The fusion rule uses a softmax function with temperature scaling, allowing for continuous adjustment of weights rather than discrete selection.

Workflow

Analyze: Compute the frequency statistic $\Delta$ and calibrate the denoising strength $\eta$ using the attacked image and reference scales.
Generate: Apply scale transformations and the shared denoiser (with strength $\eta$ ) to produce a set of candidates.
Fuse: Compute low-frequency consistency distances, apply the frequency-modulated prior, and generate softmax weights to aggregate the candidates.
Output: Clip the fused result to the valid pixel range and feed it to the target VLM.

3. Key Contributions

The paper outlines three primary contributions:

Training-Free Per-Image Strength Allocation: Unlike methods requiring fine-tuning or target-model access, FASDP adapts purification strength dynamically for each image by reweighting scale-diverse candidates from a single frozen denoiser.
Self-Referenced Frequency Statistic: A novel statistic ( $\Delta$ ) that measures the normalized change in high-frequency energy under mild transforms. It serves as a target-model-free cue to calibrate denoising strength and modulate candidate fusion without estimating true noise levels.
Low-Frequency-Consistency Fusion Rule: A closed-form fusion mechanism that aggregates candidates based on input and pairwise consistency of low-frequency components, modulated by the frequency statistic. This balances perturbation suppression with content preservation.

4. Experimental Results

The authors evaluated FASDP on three open-source VLMs (LLaVA-1.5-7B, InstructBLIP-7B, Qwen2.5-VL-7B) against three representative pixel-space attacks (VAE, HADES, Image Hijacks).

Defense Effectiveness (ASR): FASDP achieved the lowest Attack Success Rate (ASR) in the majority of tested scenarios. For instance, on Qwen2.5-VL-7B under the Image Hijacks attack, FASDP reduced ASR to 20.47%, outperforming baselines like DiffPure (24.91%) and DiffCAP (22.20%).
Benign Utility Preservation: FASDP maintained clean-image performance (VQA accuracy and CIDEr scores) closest to the undefended models. For example, on Qwen2.5-VL-7B, the VQA accuracy remained at 78.07% (compared to 81.60% for the undefended model), whereas DiffPure dropped significantly to 59.57%.
Image Fidelity: On adversarial inputs, FASDP achieved the highest SSIM (87.00) and CLIP similarity (95.45) and the lowest LPIPS (17.71) among defenses, indicating superior preservation of visual details and color fidelity compared to JPEG or DiffPure.
Robustness: The method remained effective across varying perturbation budgets (up to 64/255) and demonstrated robustness against adaptive attacks (BPDA and EOT), showing the smallest increase in ASR compared to other defenses.
Ablation Studies: Removing the frequency prior, scale diversity, or the consistency fusion rule all led to degraded performance, confirming the necessity of each module. The choice of the shared denoiser (SCUNet vs. others) had a minor impact, suggesting the framework is robust to the specific denoiser architecture.

5. Significance and Claims

The paper positions FASDP as a practical, training-free defense that addresses the specific challenge of balancing safety and utility in VLMs.

Modest Claims: The authors explicitly state that FASDP targets perturbation-based pixel-space attacks. It does not claim to defend against attacks that encode harmful intent in visible semantic content (e.g., text embedded in images) or text-only prompts.
Mechanism: The significance lies in the adaptive allocation of purification strength. By using a self-referenced frequency cue, the system avoids the "one-size-fits-all" limitation of fixed-strength defenses, allowing it to suppress strong perturbations on attacked images while preserving fine details on benign images.
Deployment: The method requires no access to the target model's internal states (gradients, embeddings) and does not require retraining, making it suitable for deployment as a pre-processing layer for existing VLM systems.

In conclusion, FASDP demonstrates that adaptive, frequency-aware purification can effectively mitigate pixel-space visual jailbreaks while maintaining the high utility of Vision-Language Models on standard tasks.