💻 computer science

REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs

This paper introduces REALM, the first unified red-teaming benchmark for physical-world Vision-Language Models, which standardizes the evaluation of diverse attack methods and defenses through a shared, physically grounded target-generation pipeline to address the fragmentation in current safety assessments.

Original authors: Yifei Zhao, Qian Lou, Mengxin Zheng

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifei Zhao, Qian Lou, Mengxin Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have built a super-smart robot assistant that can see the world and talk about it. You want this robot to drive cars, pick up objects, or help people in their homes. But before you let it loose in the real world, you need to make sure it won't get tricked into doing something dangerous, like crashing a car or grabbing the wrong tool.

This paper introduces REALM, a new "stress test" designed specifically to see how easily these robot brains can be fooled when they are dealing with real-world physical tasks.

Here is a breakdown of what the paper does, using simple analogies:

1. The Problem: The "Chatbot" Test vs. The "Real World" Test

Imagine you have a security guard (the AI).

Old Tests (Chatbot Benchmarks): These tests ask the guard, "Can you say something mean or illegal?" If the guard says "No, I won't do that," they pass. This is like testing a guard on whether they will break the rules of conversation.
The Real Problem: In the physical world, the danger isn't just about saying bad words. It's about doing the wrong thing. If a robot driving a car sees a stop sign with a tiny sticker on it, it might think it's a yield sign and keep driving. The old tests didn't catch this. They were looking for "bad words," not "bad actions."

REALM is the new test that asks: "If I trick your eyes or your instructions, will you make a physical mistake?"

2. The Solution: A Unified "Gym" for AI

Before this paper, every researcher had their own different gym, their own different weights, and their own different rules. It was impossible to compare who was actually stronger.

The REALM Gym: The authors built one giant, standardized gym. They gathered 12 different ways to trick the AI (attacks), 3 ways to protect the AI (defenses), and 13 different AI models to test.
The "Agentic" Coach: One of the hardest parts of testing is deciding what mistake to look for. If you show a picture of a car, should the AI think it's a bike? Should it think the car is driving backward?
- The paper created a special "Coach AI" (an agentic pipeline). This Coach looks at every single scene and invents a specific, realistic mistake for the AI to make. For example, in a driving scene, the Coach might decide, "Today, the AI should think the red light is green." This ensures every AI is tested against the exact same "trick," making the comparison fair.

3. The Experiments: What Happened?

The researchers ran these tests on 13 different AI models (some small, some huge) and found some surprising things:

The "Note on the Wall" Trick is the Best: The most effective way to break these robots wasn't by messing with the pixels of the image (like adding invisible noise). It was by changing the text instructions or putting a fake sign in the picture.
- Analogy: It's easier to trick a robot by whispering "Ignore the stop sign" in its ear (text injection) than by painting a tiny, invisible dot on the stop sign (visual perturbation).
The "One-Shot" Trick: Some attacks require the hacker to try hundreds of times, tweaking the image slightly each time to see what works. The researchers found that some "one-shot" attacks (trying just once) worked almost as well as the ones that tried hundreds of times. This means hackers don't need to be patient to break these systems.
Bigger Isn't Always Safer: You might think a giant, super-expensive AI (100 billion parameters) would be harder to trick than a small one. The paper found that size doesn't matter much. A huge AI was just as easily fooled as a small one if the trick was right.
Specialized Training Doesn't Help: Some AIs were specifically trained to be good at physics and reasoning. You'd think they would be smarter and harder to trick. Nope. They were just as vulnerable as the others.

4. The Defenses: Can We Stop the Tricks?

The researchers also tried three different "shields" to protect the AI:

Text Shield: One shield was great at stopping the "whispering" text tricks.
Visual Shield: Another shield was okay at stopping the "invisible dot" image tricks.
The Catch: No single shield stopped everything. If you used a text shield, the image tricks still worked. If you used a visual shield, the text tricks still worked. You need a multi-layered defense system.

Summary

REALM is the first standardized way to check if AI robots are safe in the real world. It found that:

Text tricks are currently the biggest danger.
Big models aren't automatically safe.
Specialized training doesn't make them immune to being tricked.
We need better, multi-layered defenses to keep our physical AI safe.

The paper concludes that we can't just rely on testing if AI says "bad words"; we must test if it makes "bad moves" in the physical world.

Technical Summary: REALM – A Unified Red-Teaming Benchmark for Physical-World VLMs

1. Problem Statement

Vision-language models (VLMs) are increasingly deployed as perception-reasoning backbones for embodied intelligence in safety-critical physical systems, such as autonomous vehicles and robotic manipulators. In these contexts, perception or reasoning errors can lead to unsafe physical actions. While numerous red-teaming methods exist to probe VLM vulnerabilities, the current evaluation landscape is fragmented. Existing benchmarks primarily focus on chatbot-centric safety (e.g., jailbreaking, toxic content generation) and lack standardized protocols for physically grounded functional failures.

Current challenges include:

Evaluation Fragmentation: Diverse datasets, metrics, and threat models prevent direct comparison of attack effectiveness.
Misaligned Objectives: Chatbot benchmarks measure refusal rates or toxicity, whereas physical-world VLMs require evaluation of action prediction, task completion, and spatial reasoning errors.
Lack of Unified Protocols: Existing physical-world benchmarks (e.g., PhysBench, PAI-Bench) evaluate clean performance but do not provide unified red-teaming protocols with shared attack objectives across diverse methods.

2. Methodology: The REALM Framework

The authors introduce REALM (Red-teaming Benchmark for Physical-World VLMs), the first unified benchmark designed to evaluate adversarial robustness in physical-world settings under a practical black-box threat model.

2.1 Benchmark Structure

REALM organizes evaluation along three axes:

Physical Domains: Seven domains including Driving, Manipulation, Grasping, Physics, Egocentric assistance, Diagnostics, and Scene-level Question Answering.
Task Families: Perception, Prediction, Reasoning, Evaluation, and Planning.
Failure Modes: Physically grounded output errors, such as action prediction errors, task success errors, spatial reasoning errors, and embodied planning errors.

The dataset comprises 832 test cases derived from existing physical-AI benchmarks, expanded to 9,984 adversarial test cases across 12 red-teaming methods and 13 victim VLMs.

2.2 Agentic Target-Generation Pipeline

A core innovation of REALM is the agentic target-generation pipeline, which addresses the difficulty of defining fixed attack objectives for scene-specific physical tasks.

Process: Given a source image, question, and ground-truth answer, a reasoning VLM agent analyzes the scene to identify a plausible failure mode (e.g., misinterpreting a vehicle's maneuver).
Output: The pipeline generates a scenario-specific target answer ( $a^{tar}_i$ ) and a target image ( $I^{tar}_i$ ) that visually instantiates the intended misperception.
Utility: These shared targets allow diverse red-teaming methods (visual perturbations, prompt injection, etc.) to be evaluated against the same scenario-specific adversarial goals, ensuring fair comparison.

2.3 Threat Model and Evaluation

Threat Model: A realistic black-box setting. Neither the attack nor defense methods have access to the victim model's parameters, gradients, or internal states.
Attack Generation: Adversarial examples are generated offline using surrogate models or external generators and transferred to the victim model for one-shot evaluation.
Metrics: The primary metric is Attack Success Rate (ASR), defined as the rate at which an adversarial input causes the VLM to produce an incorrect response reflecting a task-relevant physical error. Clean accuracy is also reported for baseline comparison.

2.4 Evaluated Components

Victim Models: 13 VLMs (7B to 100B+ parameters), including proprietary models (Gemini-3-Flash, GPT-4.1-Mini) and open-source families (Qwen3.5, Qwen3-VL, Cosmos-Reason).
Red-Teaming Methods: 12 methods covering:
- Visual Perturbations: Gradient-based (FOA, M-Attack, V-Attack), Co-optimization (Chain-of-Attack), and Diffusion-based (AdvDiffVLM).
- Physical Constraints: PhysPatch (localized patches).
- Generative/Editing: AdvEDM, ImageMix.
- Injection: Typographic (FigStep), Prompt (PromptInject), and Single-pass (AnyAttack).
Defenses: 3 model-agnostic preprocessing defenses: PAD (patch removal), FreqPure (frequency purification), and BlueSuffix (text/image purification).

3. Key Results

3.1 Attack Effectiveness

Text and Typographic Injection: These attacks induce the most failures. PromptInject achieved the highest average ASR (64.8%), followed by FigStep (49.4%). This indicates that VLMs remain highly vulnerable to manipulation of the text and auxiliary instruction channels.
Visual Perturbations: Among visual attacks, Chain-of-Attack (CoA), which co-optimizes perturbations with dynamically regenerated captions, yielded the strongest transfer (54.6% ASR), outperforming standard gradient-based attacks. This suggests multimodal co-optimization exploits cross-modal alignment better than visual-feature-only optimization.
Efficiency: AnyAttack, a single-pass method, achieved 49.9% ASR, approaching the performance of iterative methods (like FOA at 48.1%) at a significantly lower computational cost.
Physical Constraints: Spatially restricted perturbations (PhysPatch) showed lower transferability (33.0% ASR) compared to full-image perturbations, indicating that physical plausibility constraints make black-box transfer more difficult.

3.2 Model Vulnerability

Scale vs. Robustness: Increasing model scale (e.g., Qwen3.5 from 9B to 122B) improves clean accuracy but does not guarantee adversarial robustness. ASR for specific attacks (e.g., PromptInject) remained nearly unchanged or even increased with scale.
Domain Specialization: Models post-trained for physical reasoning (Cosmos-Reason) did not exhibit superior robustness; in fact, they showed high vulnerability, with Cosmos-Reason2-8B reaching 76.8% ASR under PromptInject.
Proprietary vs. Open Source: Proprietary models (Gemini, GPT) demonstrated vulnerability levels comparable to open-source models with similar clean accuracy, suggesting closed-source deployment does not inherently ensure robustness.
Domain Variance: Vulnerability varies significantly by domain. Manipulation and Driving tasks showed the highest median ASR, while Scene QA showed the lowest.

3.3 Defense Analysis

Channel Specificity: Defenses are highly channel-specific. BlueSuffix (text purification) significantly reduced PromptInject ASR by 21.7% on average. FreqPure (frequency filtering) reduced gradient-based attacks (FOA) by 5.8%.
Limitations: No single preprocessing defense generalized across all attack paradigms. For instance, visual defenses (PAD, FreqPure) failed to mitigate text-injection attacks, and none of the defenses affected FigStep.

4. Significance and Contributions

The paper claims REALM provides the first unified framework for comparing red-teaming methods in physical-world VLMs. Its significance lies in:

Unified Benchmark: Integrating 12 attack methods, 3 defenses, and 13 VLMs under a shared protocol, enabling direct comparison of effectiveness and coverage.
Agentic Target Generation: Solving the problem of scenario-specific objective alignment, allowing diverse attacks to be evaluated against physically grounded, scenario-consistent failure modes.
Empirical Insights: Establishing that text-channel vulnerabilities are currently the most critical failure point, that multimodal co-optimization enhances visual transfer, and that model scale alone is insufficient for robustness.

The authors conclude that future work must move beyond chatbot-oriented content safety to develop dedicated benchmarks and defenses for functional robustness in grounded physical tasks.