Technical Summary: GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization

Problem Statement

Diffusion Vision-Language Models (dVLMs) represent a significant shift from traditional autoregressive generative paradigms, utilizing non-causal, iterative denoising processes for multimodal tasks. While autoregressive models have undergone extensive safety alignment to mitigate adversarial risks, the security landscape of dVLMs remains largely unexplored.

The paper identifies a critical gap: existing jailbreak methodologies, specifically Fixed Prefix Optimization (FPO) (e.g., forcing a model to start with "Sure, here is"), are largely ineffective against dVLMs. This ineffectiveness stems from unique refusal patterns inherent to diffusion architectures:

1. Immediate Refusal: The model detects risks and generates termination tokens (e.g., <end>) in a "back-to-front" manner, halting generation immediately.
2. Progressive Refusal: The model initially generates affirmative content but, during the iterative refinement process, pivots to a refusal as unsafe semantics emerge.

Conventional FPO attacks, which optimize local prefixes, fail because the global refinement nature of diffusion models acts as a "corrector," eventually overriding local prefix manipulations with a progressive refusal. The paper posits that a new attack strategy is required to bypass these global probabilistic defenses.

Methodology

1. Heuristic Inference and Theoretical Basis

The authors draw an analogy between image diffusion and text diffusion. In image generation, early denoising steps recover low-dimensional semantic structures that dictate the final output. Similarly, in dVLMs, the initial generation steps (low timesteps) largely determine the response polarity. The global bidirectional attention predictions at these early stages constrain the subsequent generative trajectory.

2. Global Probability Optimization (GPO)

To exploit this vulnerability, the authors propose Global Probability Optimization (GPO), a strategy designed specifically for masked diffusion models. Unlike FPO, which optimizes a fixed prefix, GPO manipulates the global probability distribution of the entire generated sequence.

The GPO loss function ($L_{GPO}$) consists of two components:

• Global Loss ($L_{global}$): Minimizes the probability of generating negative primitives (refusal tokens like <end>, <eos>, or refusal phrases) across all positions in the sequence, particularly at the initial and final stages of early timesteps. This aims to suppress both Immediate and Progressive Refusal patterns.
• Anchor Loss ($L_{anchor}$): Maximizes the probability of specific "connective" positive tokens (anchors) at intermediate positions. These anchors serve as stable semantic pivots that accumulate into a coherent positive response, preventing the model from drifting into a progressive refusal.

The total loss is defined as:
$$L_{GPO} = \lambda_1 L_{global} + \lambda_2 L_{anchor}$$

3. GPO-V Framework

Building on GPO, the authors introduce GPO-V, the first jailbreak framework tailored for dVLMs.

• Mechanism: GPO-V applies adversarial perturbations ($\delta$) to the input image (or its latent representation) to steer the model's output distribution.
• Optimization: The perturbation is optimized to minimize $L_{GPO}$ while adhering to a perturbation bound ($\|\delta\|_\infty < \epsilon$).
• Flexibility: The framework supports optimization in both pixel space and latent space (using encoders like SigLIP or DINOv2), with the authors noting that latent-space optimization is adopted for efficiency but yields comparable results.

Key Contributions

1. Identification of Refusal Patterns: The paper is the first to systematically identify and categorize two distinct refusal behaviors in diffusion language models: Immediate Refusal and Progressive Refusal, revealing the underlying defense mechanisms of dVLMs.
2. GPO Strategy: The proposal of Global Probability Optimization, a novel strategy that bypasses safety guards by globally suppressing negative token probabilities and optimizing anchor losses, contrasting with the prefix-based optimization of autoregressive models.
3. GPO-V Framework: The development of the first visual-modality jailbreak framework for dVLMs. It demonstrates that dVLMs are inherently fragile against global probabilistic attacks, despite their apparent robustness against FPO.

Experimental Results

The authors evaluated GPO-V on two state-of-the-art masked diffusion vision-language models: LLaDA-V and LaViDA, using the AdvBench and JailBreakBench datasets.

• Attack Success Rate (JSR): GPO-V achieved a jailbreak success rate of up to 93.2% on LLaDA-V and 88.9% on LaViDA. In contrast, traditional FPO-based baselines (e.g., GCG) achieved negligible success rates (often < 2%).
• Transferability: The adversarial perturbations synthesized by GPO-V demonstrated high cross-model transferability. Attacks optimized on a source model successfully compromised victim models with heterogeneous architectures (e.g., transferring from LLaDA-V to LaViDA) with success rates exceeding 65%, indicating the capture of model-agnostic vulnerabilities.
• Robustness: The method remained effective across different generation configurations, including varying block lengths and the use of Fast-dLLM acceleration techniques.
• Ablation Studies: Removing either the global loss or the anchor loss significantly reduced attack effectiveness, confirming that both the suppression of refusal patterns and the stabilization of positive anchors are necessary for success.

Significance and Claims

The paper claims that its findings underscore a critical security gap in non-sequential generative architectures. The primary significance lies in:

• Challenging Perceived Robustness: The results demonstrate that the perceived resilience of dVLMs against conventional jailbreaks is deceptive. The unique "Progressive Refusal" mechanism, which acts as a defense against local prefix attacks, is itself a vulnerability when targeted by global probability optimization.
• Necessity for Re-evaluation: The authors argue that current defense paradigms, largely optimized for causal (autoregressive) decoding, are insufficient for the unique generative manifold of diffusion models.
• Call to Action: The work necessitates an immediate and fundamental re-evaluation of safety alignment for dVLMs to address the risks of diffusion-based generation, urging the research community to develop robust defenses against these emerging non-sequential threats.

The paper concludes that GPO-V exposes intrinsic vulnerabilities in the diffusion process itself, suggesting that safety alignment must evolve to account for global probabilistic dynamics rather than just sequential token prediction.