Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios

This paper addresses the limitations of current risk-oriented methods in constructing multimodal safety datasets by proposing a novel image-oriented self-adaptive pipeline that automatically generates a 35k real-world safety dataset and introduces a standardized evaluation metric to validate its effectiveness across various tasks.

The Gist

Imagine you are teaching a robot how to be a good guardian. You want it to spot danger before it happens. But here's the problem: the robot has been trained mostly on cartoons of danger.

If you show the robot a picture of a cartoon fire and ask, "Is this safe?" it says, "No, fire is bad!" It's easy to spot danger when it's screaming at you from a comic book.

But in the real world, danger is quiet. It's a safe-looking picture of a library paired with a safe-looking sentence like, "I want to start a fire to keep warm." Alone, the library is fine. Alone, the sentence is fine. But put them together? Disaster. The robot, trained only on obvious cartoons, misses the trap completely.

This paper introduces a new way to build a "training gym" for these AI guardians, called RMS (Real-World Multimodal Safety). Here is how they did it, explained simply:

1. The Problem: The "Cartoon" Trap

Current safety tests are like a driving test where the instructor only puts a giant red "STOP" sign on the road. The AI learns to stop for red signs. But in real life, the danger is a slippery patch of ice that looks like normal pavement, or a car that looks like a toy but is actually speeding.

• Old Way: Create fake, obvious dangers (synthetic images) and tell the AI, "This is bad."
• The Flaw: The AI gets good at spotting the fake signs but fails when the danger is hidden in a normal, everyday scene.

2. The Solution: The "Lego" Approach (Image-Oriented)

Instead of starting with a scary story and trying to find a picture to match it, the authors started with real-world photos (like a picture of a cliff, a library, or a kitchen) and asked a smart AI assistant: "What if someone said something harmless here that would actually be dangerous?"

Think of it like a Lego set:

• Piece A (The Image): A picture of a high cliff. (Safe on its own).
• Piece B (The Text): A sentence saying, "I want to jump." (Safe on its own).
• The Combination: When you snap them together, you get a suicide risk.

The authors built a machine that automatically snaps millions of these "safe" pieces together to find the hidden "dangerous combinations." They call this Information Complementarity. It's like realizing that a "knife" is safe, and "cooking" is safe, but "stabbing" is not. The danger comes from the mix.

3. The Result: A Massive "Trap" Dataset

They used this method to build a dataset of 35,000 scenarios.

• The Images: All taken from the real world (no cartoons).
• The Text: All harmless sentences.
• The Trap: The combination creates a hidden risk (like falling, fire, or self-harm).

They also created two types of "answers" for the AI to learn from:

1. The "Bad" Answer: An AI that ignores the trap and says, "Great idea! Go jump!" (This teaches the AI what not to do).
2. The "Good" Answer: An AI that spots the trap and says, "Wait, that cliff is dangerous! Let's find a safer way." (This teaches the AI what to do).

4. The New Scorecard

The authors realized we didn't have a good way to measure if a safety dataset was actually good. So, they invented a new test:

• The "Teacher" Test: Take a safety dataset, use it to train a "Safety Judge" AI, and then see how well that Judge performs on other safety tests.
• The Result: The AI trained on their new "Real-World" dataset became a much better judge than those trained on old "Cartoon" datasets. It learned to see the ice on the road, not just the red stop signs.

5. The Big Discovery

When they tested current famous AI models (like GPT-4o or Gemini) on this new dataset, they failed miserably.

• Most models couldn't see the danger. They saw a picture of a cliff and a text about "jumping" and thought, "Oh, a fun park!" or "A great adventure!"
• They only realized it was dangerous when the text was explicitly screaming "I want to kill myself." They missed the subtle, real-world risks.

Summary

This paper is like building a driving simulator that doesn't just show you red lights and stop signs. Instead, it shows you a rainy day, a slippery road, and a distracted driver, and asks, "What happens if you don't slow down?"

By training AI on these subtle, real-world combinations of safe images and safe text, the authors hope to build guardians that can actually protect us in the messy, complex real world, not just in a cartoon.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) are rapidly evolving, yet their safety mechanisms face significant challenges in real-world scenarios. The authors identify two critical limitations in current safety dataset construction and evaluation:

• Over-reliance on Synthetic/Risk-Oriented Data: Existing benchmarks often rely on synthetic images or preset risks. They fail to capture the complexity of Real-World Multimodal Safety (RMS) scenarios, where risks emerge from the complementarity of independently safe modalities (e.g., a safe image of a cliff + a safe text about "jumping" = suicide risk).
• Lack of Unified Evaluation Metrics: There is no standardized metric to evaluate the effectiveness of safety datasets. Current methods do not adequately prove whether a dataset actually improves a model's ability to judge safety in complex, real-world contexts.
• Poor Model Performance: Preliminary observations suggest that even state-of-the-art MLLMs struggle to identify latent risks when safe images and safe texts are combined, often failing to detect unsafe outcomes.

2. Methodology: Image-Oriented Self-Adaptive Construction

The paper proposes a novel Image-Oriented Self-Adaptive Pipeline to construct the RMS Dataset. The core concept is Information Complementarity: situations where combining safe information from different modalities creates a latent unsafe outcome.

The pipeline consists of four main stages:

A. Pattern Generation (Inspiration Phase)

• Source: Uses real-world images from the COCO dataset as the "inspiration dataset."
• Process: An AI assistant (Gemini-1.5-Flash) analyzes images to extract latent unsafe information (e.g., a cliff implies "falling").
• Complementarity: The AI generates text descriptions of activities that, when paired with the image, create a risk (e.g., "I want to jump").
• Safety Filtering: The text is rewritten to be safe in isolation (e.g., removing explicit mentions of the image context) but remains dangerous when combined with the image.
• Categorization: The AI identifies the specific risk category (e.g., Suicide, Property Damage).

B. Data Augmentation (Scalability Phase)

• Keyword Extraction: Keywords are extracted from the generated image-text pairs.
• Search & Filter: These keywords are used to search the LAION-5B dataset for similar real-world images.
• Verification: An AI assistant (InternVL2.5-78B) filters these images to ensure they are authentic, real-world, and safe in isolation.
• Pairing: New text is generated based on the keywords and paired with the filtered images to create a scalable dataset.

C. Guidance Response Generation

• Unsafe Responses: The AI is prompted to encourage the behavior described in the text (ignoring the image context), generating an "unsafe" response.
• Safe Responses: The AI is prompted with the image-text pair and the identified risk, generating a response that identifies the danger and offers safety advice.

D. Safety Review

• A rigorous review process using InternVL2.5-78B ensures:
  1. Images are real-world and safe.
  2. Text is safe in isolation.
  3. The combined image-text pair creates the intended risk.
  4. Guidance responses are safe.
• Manual sampling (500 points) confirmed 100% compliance.

3. Key Contributions

A. The RMS Dataset

• Scale: Contains 35,610 image-text pairs with guidance responses.
• Structure: Organized into 12 main categories (e.g., Suicide/Self-harm, Illegal, Discrimination) and 39 fine-grained scenarios.
• Uniqueness: Unlike previous benchmarks, every single image and text is safe individually; the risk is purely emergent from their combination. All images are sourced from real-world scenarios, not synthetic generation.

B. New Evaluation Metric: "Fine-tuned Model as Metric"

The authors propose a standardized metric to evaluate safety datasets:

1. Fine-tune a safety judge model (e.g., Llama-3.2-11B-vision) on a specific dataset.
2. Evaluate this fine-tuned model's performance on other independent safety datasets.
3. Hypothesis: A better safety dataset will produce a judge model that generalizes better to unseen safety scenarios.

C. Standardized Categorization

The paper defines a hierarchical taxonomy of real-world safety risks based on information complementarity, moving beyond simple "unsafe image" or "unsafe text" classifications.

4. Experimental Results

A. Incremental Experiments

• As the scale of the inspiration dataset increased, the image-oriented method adaptively discovered more risk categories.
• Fine-tuning models on larger RMS subsets improved safety judgment accuracy across other datasets, validating the scalability of the approach.

B. Baseline MLLM Performance (The "RMS Test")

• Finding: Current mainstream MLLMs (including GPT-4o, Gemini-1.5, Llama-3.2) perform poorly on the RMS test set.
• Metrics: Many models have an accuracy for identifying unsafe responses worse than random selection (50%).
• Reasoning: Models fail to detect the latent risk when inputs are individually safe. For example, a model might encourage a user to "jump" because the text alone seems harmless, ignoring the cliff in the image.
• Safety Rate: Even the best-performing model (Gemini-1.5-Flash) achieved only a 22% safety rate when directly responding to RMS inputs.

C. Evaluation Metric Validation

• Models fine-tuned on the RMS dataset significantly outperformed models fine-tuned on other benchmarks (like VLGuard, Ch3Ef, MSSBench) when tested on those same external datasets.
• This confirms that the RMS dataset effectively enhances the general safety judgment capabilities of MLLMs.

5. Significance and Impact

• Paradigm Shift: Moves safety research from "risk-oriented" (finding bad images/text) to "complementarity-oriented" (finding hidden risks in safe combinations).
• Real-World Relevance: By using real-world images, the dataset better reflects the actual deployment environment of MLLMs, where users often combine safe-looking photos with ambiguous text.
• Benchmarking Standard: The proposed "Fine-tuned Model as Metric" offers a new, rigorous way to compare the quality of safety datasets, addressing the lack of unified evaluation standards.
• Safety Gap: The results highlight a critical vulnerability in current MLLMs: they are easily "jailbroken" or misled by the subtle interplay of safe modalities, necessitating the use of datasets like RMS for robust alignment.

Conclusion: The paper successfully demonstrates that an image-oriented, self-adaptive approach can construct a large-scale, high-quality dataset that exposes and addresses the hidden safety risks in real-world multimodal interactions, providing a new foundation for training safer MLLMs.