Technical Summary: Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models

Problem Statement

Large Vision-Language Models (VLMs) have demonstrated exceptional capabilities in multimodal tasks but face significant challenges when deployed in safety-critical domains. Existing safety fine-tuning methods, primarily relying on Textual Supervised Fine-Tuning (SFT) or single-image Multimodal SFT (e.g., VLGuard), exhibit critical bottlenecks:

1. Safety Visual Reasoning Gap: Current methods lack the ability to reason about safety within visual contexts. They often fail to detect unsafe intent when it arises from the interaction between multiple images and a benign text query, rather than from explicit harmful elements in a single image.
2. Over-Prudence and Trade-offs: Fine-tuning on existing datasets often leads to "over-conservatism," where models refuse to answer benign queries (e.g., responding "I'm sorry" to safe inputs). Conversely, methods that attempt to reduce refusals often fail to generalize to challenging safety scenarios, resulting in high Attack Success Rates (ASR) on complex tasks.
3. Limitations of Single-Image Paradigms: Most safety benchmarks and training data rely on single-image inputs. As VLMs advance in multi-image understanding, the safety gap widens because existing safeguards cannot effectively interpret the contextual risks formed by combining multiple visual inputs.

Methodology

To address these limitations, the authors propose a novel framework centered on the Multi-Image Safety (MIS) dataset and the MIRage (Multi-Image Reasoning Safety) fine-tuning approach.

1. The MIS Dataset Construction

The MIS dataset is the first multi-image safety dataset designed to evaluate and improve safety-related visual perception and reasoning. It consists of:

• Training Set (4k samples): Multi-image pairs linked by text instructions that create unsafe intents (e.g., using a tool from Image A to manipulate an object in Image B). Crucially, the text instructions are detoxified (neutral), while the safety risk emerges from the visual combination.
• Test Set (2185 samples): Divided into three subsets:
  • MIS-easy: Contains explicit unsafe elements in images.
  • MIS-hard: Contains only benign images where the unsafe intent is derived solely from the relationship between the images and the query.
  • MIS-real: Uses real-world images retrieved from datasets to test generalization.
• Construction Pipeline: The dataset is generated via a four-step automated pipeline involving:
  1. Harmful Element Extraction: Using LLMs (Qwen2.5, InternVL) to extract unsafe concepts from existing benchmarks.
  2. Text Instruction Generation & Detoxification: Generating harmful queries involving two objects, then refining them into neutral instructions (e.g., replacing "smuggle artifacts" with "transport objects in the first image").
  3. Auto-Refinement T2I Generation: Using Stable Diffusion 3.5 Large to generate images, refined by VLMs to ensure alignment with the text.
  4. Multi-Expert Filtering: Human experts and GPT-4o filter and classify data into training, easy, hard, and real subsets.

2. Safety Chain-of-Thought (CoT) Labels

A core innovation is the construction of Safety CoT labels for the training set. Instead of simple rejection responses (e.g., "I cannot do that"), the labels guide the model to:

1. Perform Visual Perception: Identify objects in both images.
2. Execute Visual Reasoning: Analyze the relationship between the images and the text to deduce potential harmful intent.
3. Generate a Safe Response: Provide a helpful, context-aware refusal or warning that explains why the request is unsafe, rather than a blanket refusal.

3. The MIRage Fine-Tuning Approach

The authors introduce MIRage, a fine-tuning strategy that utilizes the MIS training set.

• Data Composition: The training set includes 4k safety-related multi-image samples and 500 general QA samples (11% of the total), significantly lower than the 33% general data ratio used in prior works like VLGuard.
• Training Objective: The model is trained to integrate visual perception and reasoning logic into its safety responses, aiming to balance helpfulness and harmlessness without sacrificing general capabilities.

Key Contributions

1. Identification of Bottlenecks: The paper analyzes existing safety fine-tuning methods, revealing that their failure on challenging tasks (like MSSBench) stems from a lack of safety visual reasoning and the use of simplistic rejection-based labels that cause over-prudence.
2. MIS Dataset: The introduction of the first multi-image safety dataset, featuring a training split for enhancing visual reasoning and three-level test splits (easy, hard, real) for rigorous evaluation.
3. MIRage Framework: A novel fine-tuning paradigm that leverages Safety CoT labels. It demonstrates that improving visual reasoning capabilities allows models to handle complex multi-image safety scenarios effectively.
4. Empirical Validation: Comprehensive experiments showing that MIRage outperforms both open-source and API-based models on multi-image safety tasks while preserving or slightly improving general capabilities.

Experimental Results

The authors evaluated MIRage primarily on InternVL2.5-8B, comparing it against baselines like Textual SFT, VLGuard (P and R variants), and various state-of-the-art VLMs (e.g., Qwen2-VL, LLaVA-OV, GPT-4o, Gemini-1.5-pro).

• Safety Performance:

  • On the MIS test set, MIRage reduced the Attack Success Rate (ASR) to near-zero levels (e.g., 0.24% on MIS-easy, 0.20% on MIS-hard, 0.00% on MIS-real) for InternVL2.5-8B, compared to ~80-90% ASR for baseline models.
  • It significantly increased the Reasoning Success Rate (RSR), indicating models are successfully identifying risks through reasoning rather than random refusal.
  • MIRage outperformed baselines on challenging benchmarks like MSSBench (Unsafe scenarios) and SIUO (Safety Inputs, Unsafe Outputs), where existing methods often failed or exhibited over-prudence.

• General Capabilities:

  • Unlike VLGuard and Textual SFT, which often degraded general performance (especially on multi-image benchmarks like MMT-Bench and MuirBench), MIRage preserved and slightly enhanced general capabilities.
  • Fine-tuning with MIS increased average accuracy by 0.83% across five general benchmarks (Q-Bench, MMStar, MMMU, MuirBench, MMT-Bench).
  • The method achieved this with only 11% general data in the training mix, demonstrating that safety reasoning logic can inherently support general instruction following.

• Generalization:

  • Ablation studies showed that safety capabilities learned via MIRage generalize to unseen safety categories (e.g., Privacy and Self-Harm) and even transfer to text-only jailbreak scenarios (AdvBench).

Significance and Claims

The paper claims that the primary bottleneck in current VLM safety is not a lack of data, but a lack of safety visual reasoning and the use of oversimplified training labels. By shifting the paradigm from single-image rejection to multi-image reasoning with Chain-of-Thought logic, the authors demonstrate that:

1. Safety and Helpfulness are not mutually exclusive: Models can be trained to be highly safe in complex scenarios without becoming overly conservative or losing general utility.
2. Multi-Image Context is Critical: As VLMs evolve to handle multi-image inputs, safety evaluation and training must evolve beyond single-image paradigms to address contextual risks.
3. Reasoning is the Key: Incorporating reasoning logic into safety labels is more effective than simple rejection, enabling models to distinguish between safe and unsafe scenarios in nuanced, multi-modal contexts.

The authors conclude that their work lays the groundwork for future research into more sophisticated safety data and strategies specifically designed for the multi-image setting, addressing a critical gap in the deployment of safe VLMs.