💬 NLP

Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?

This paper introduces Distract-Bench, a new benchmark demonstrating that Reasoning Vision-Language Models are significantly more vulnerable to semantic visual distractions—meaningful but irrelevant cues—than to traditional perceptual corruptions, as these distractions often mislead the models' reasoning processes despite correct visual perception.

Original authors: Yizheng Sun, Mochuan Zhan, Yanan Ma, Jia Tong See, Yifan Wang, Ziyi Wang, Hao Li, Yang Cui, Wenhao Cai, Jingyu Sun, Chenghua Lin, Riza Batista-Navarro, Jingyuan Sun

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yizheng Sun, Mochuan Zhan, Yanan Ma, Jia Tong See, Yifan Wang, Ziyi Wang, Hao Li, Yang Cui, Wenhao Cai, Jingyu Sun, Chenghua Lin, Riza Batista-Navarro, Jingyuan Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Smart Students vs. Distracting Classrooms

Imagine you have a very smart student (a Reasoning Vision-Language Model or VLM). This student is great at looking at a picture and a question, thinking through the problem step-by-step, and giving the right answer. They are like a math whiz who can solve complex geometry problems just by looking at a diagram.

For a long time, researchers tested how "tough" these students were by making the classroom messy. They would:

Blur the blackboard (blur).
Turn off the lights (noise).
Shake the camera (weather effects).

This is called Perceptual Robustness. It asks: "Can the student still see the answer if the view is fuzzy?"

This paper introduces a new, sneakier problem.

Instead of making the view fuzzy, the researchers kept the blackboard crystal clear. But, they added a distractor: a piece of paper taped to the board with a true fact that has nothing to do with the question.

The Question: "How many apples are in the basket?"
The Image: A basket with 3 apples.
The Distractor: A bright red sign taped next to the basket that says, "There are 5 oranges in the next room." (This is a true fact, but it's irrelevant).

The researchers call this Semantic Distraction. They wanted to see: If the student can see clearly, will they get tricked by the irrelevant information?

The Experiment: Distract-Bench

The team built a test called Distract-Bench.

The Setup: They took 506 real-world questions and images (like math problems, charts, and daily scenes).
The Trick: They used AI and human experts to add "distractors" to the images. These distractors were:
- Factually True: The sign really did say "5 oranges."
- Visually Plausible: It looked like it belonged on the board.
- Totally Irrelevant: It didn't help answer the question about the apples.
- Answer-Preserving: The correct answer (3 apples) didn't change.

They then tested 10 different models (some are "reasoning" models that think step-by-step, and some are standard models) to see how they handled these tricks.

The Surprising Findings

The results were counter-intuitive and revealed a hidden weakness in "smart" models.

1. The "Blur" Test vs. The "Distractor" Test

When the image was blurry (Perceptual Robustness): The "smart" reasoning models behaved almost exactly like their "dumber" base versions. If the base model couldn't see through the blur, the smart model couldn't either. They inherited the same visual limitations.
When the image had a distractor (Semantic Robustness): The "smart" reasoning models actually performed worse than the base models! Even though they were better at thinking, they were more easily tricked by the irrelevant facts.

Analogy: Imagine a student who is great at math but gets distracted by a fly buzzing around the room. A simpler student might just ignore the fly and focus on the numbers. The "smart" student, however, starts thinking, "Wait, is that fly a symbol? Does it mean I should add 1 to the answer?" Their ability to over-think actually hurt them.

2. The Reasoning Trap
The researchers looked at why the models failed. They found that the "smart" models didn't just guess wrong; they incorporated the distractor into their logic.

The Failure Mode: The model would see the red sign saying "5 oranges," treat it as evidence, and build a long, logical chain of reasoning based on that wrong clue.
The Result: They would produce a very confident, well-explained, but completely wrong answer. The "reasoning" part of the model amplified the mistake.

3. The "Harmful Reference"
The study measured how often the models mentioned the distractor in their final answer.

Smart models mentioned the irrelevant facts much more often than the base models.
When they got the answer wrong, they were almost always referencing the distractor as proof for their mistake.

The Conclusion

The paper concludes that being good at reasoning doesn't make you immune to distractions. In fact, for these AI models, the ability to generate long chains of thought can sometimes make them more vulnerable to irrelevant information.

Old View: We only need to worry if the image is blurry or noisy.
New View: We also need to worry if the image contains "too much truth." If a model sees a fact that is true but irrelevant, it might grab onto it and let it derail its entire thought process.

The Takeaway: To make these AI models reliable in the real world, we can't just teach them to see better; we have to teach them to ignore things that don't matter, even if those things look important and are factually true.

(Note: This explanation is based strictly on the findings presented in the paper. The authors do not discuss specific clinical applications or future commercial uses in this text, so none are included here.)

Technical Summary: Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?

Problem Definition

While Reasoning Vision-Language Models (VLMs) have demonstrated strong performance on complex multimodal tasks, their reliability in real-world applications remains unproven. Existing robustness evaluations primarily focus on perceptual robustness, testing models against input corruptions such as noise, blur, and weather effects. These corruptions degrade the visibility of evidence but do not introduce new semantic content.

This paper identifies a critical, underexplored failure mode: semantic visual distractions. In real-world scenarios, inputs often contain factually correct but task-irrelevant visual cues (e.g., extra objects, misleading annotations, or contextual details). A model may correctly perceive the relevant evidence yet attend to these plausible but irrelevant cues, incorporate them into its reasoning chain, and propagate the error to an incorrect final answer. This is particularly concerning for reasoning-enhanced VLMs, where long-form reasoning processes can amplify the impact of initially irrelevant details, turning them into coherent but incorrect reasoning paths.

Methodology

1. Distract-Bench Benchmark

To address this gap, the authors introduce Distract-Bench, a human-verified benchmark designed to evaluate VLM robustness to answer-preserving semantic distractions. The dataset construction follows a four-stage pipeline:

Raw Sample Collection: 800 image-question pairs were collected from established reasoning-heavy benchmarks (MathVision, MathVista) and general-domain benchmarks (MMMU).
Expert-Designed Seed Distractors: Human annotators created 100 seed examples by injecting semantic distractors into images. These distractors were required to be: (1) semantically obvious, (2) factually correct and visually plausible, (3) irrelevant to the query (preserving the ground-truth answer), and (4) capable of leading to plausible wrong answers.
Scalable Synthetic Expansion: Using the seed examples as a knowledge base, the authors leveraged GPT-5.5 to generate edit instructions for remaining samples and GPT-Image2 to synthesize the edited images. This process ensured the original content and answer remained unchanged while adding the distractor.
Human Quality Verification: Experts verified all generated samples against the four criteria, resulting in a final dataset of 506 high-quality samples spanning diverse domains (charts, math, daily scenes, medical, etc.).

2. Experimental Setup

The study evaluated 10 VLMs, including:

Open-Source Reasoning Models: R1-OneVision, Vision-R1, MMR1, MM-Eureka, OpenVLThinker, and Qwen3-VL-Thinking.
Corresponding Non-Reasoning Base Models: Qwen2.5-VL (Base), Qwen3-VL (Base), etc., to enable controlled paired comparisons.
Closed-Source Models: GPT-4o and GPT-5.

Models were tested under three conditions:

Clean Inputs: Baseline performance.
Conventional Vision Corruptions: 20 types of perceptual degradations (noise, blur, weather, etc.).
Distract-Bench: Inputs with injected semantic distractors.

3. Metrics

Relative Robustness (RR): The fraction of clean performance preserved after perturbation ( $RR = P_{O}/P_{I}$ ).
Retention: The rate at which a model remains correct on samples it answered correctly in the clean setting.
Distraction Reference Rate (DRR): Frequency of explicit mentions of the injected distractor in model outputs.
Harmful Reference Rate (HFR): Frequency of distractor mentions associated with incorrect answers.

Key Results

1. Distinct Failure Modes

The study reveals that semantic distraction exposes a robustness failure mode distinct from conventional vision corruption.

Vision Corruption: Reasoning VLMs largely inherit the robustness patterns of their non-reasoning base models. Under perceptual degradation, the relative robustness and error overlap between base and reasoning variants are highly aligned (e.g., mean Pearson correlation of 0.978 for Qwen2.5-VL family).
Semantic Distraction: Reasoning VLMs show consistently lower robustness compared to their base models when faced with semantic distractions. For instance, in the Qwen2.5-VL family, the base model retained 92.1% relative robustness on Distract-Bench, while reasoning variants dropped to 84.4%–90.3%. This indicates that reasoning capabilities do not confer immunity to irrelevant semantic cues; in fact, they may exacerbate sensitivity.

2. Sensitivity to Task-Irrelevant Cues

Despite strong performance on clean data, reasoning VLMs are highly sensitive to task-irrelevant semantic cues. The accuracy drop for reasoning models on Distract-Bench (8.1–12.8 percentage points) was significantly larger than for their non-reasoning counterparts (6.3 percentage points). This suggests that reliable reasoning under clean inputs is insufficient to ensure robust evidence selection when plausible but irrelevant visual cues are present.

3. Distractors Enter the Reasoning Process

Output-level analysis confirms that failures are not random answer flips but are driven by the integration of distractors into the reasoning chain.

Higher Reference Rates: Reasoning models referenced distractors more frequently than base models (higher DRR).
Harmful Integration: The Harmful Reference Rate (HFR) was consistently higher for reasoning models. In failure cases, models frequently treated the distractor as valid evidence, using it to justify incorrect conclusions.
Qualitative Evidence: Case studies (e.g., geometry problems with injected angle annotations) showed that models would explicitly cite the distractor (e.g., "We're told that angle ODC is 30 degrees...") and derive a wrong answer based on it, whereas base models were more likely to ignore the cue or mention it harmlessly.

Significance and Claims

The paper claims that Distract-Bench reframes the evaluation of robustness for reasoning VLMs. The authors argue that the field must shift focus from purely degraded perception (can the model see the evidence?) to semantic distraction (can the model ignore irrelevant but plausible evidence?).

Key claims include:

Stronger reasoning does not equal stronger reliability: Enhancing reasoning capabilities via reinforcement learning or supervised traces does not necessarily make models more robust to distribution shifts involving semantic noise; it may even make them more susceptible to "reasoning traps."
The "Reasoning Amplification" Effect: Long-form reasoning can amplify the impact of irrelevant evidence. Once a distractor is selected as part of the evidence chain, the reasoning process can construct a coherent but incorrect narrative, leading to confident wrong answers.
Necessity of New Evaluation: Reliable real-world VLMs require evaluation under misleading semantic distractions, not just perceptual corruptions.

The authors conclude that developing VLMs that are both reasoning-capable and robust remains a critical future direction, potentially requiring improved evidence selection mechanisms, distraction-aware training, or inference-time verification. They explicitly state that their work is diagnostic, revealing these risks without proposing a specific mitigation method.