🤖 machine learning

VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning

This paper introduces VisReason, a large-scale dataset of 489K annotated examples with human-like, multi-round rationales, and its expert-curated subset VisReason-Pro, which significantly enhance the step-by-step visual reasoning, interpretability, and generalization capabilities of multimodal large language models.

Original authors: Lingxiao Li, Yifan Wang, Xinyan Gao, Chen Tang, Xiangyu Yue, Chenyu You

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lingxiao Li, Yifan Wang, Xinyan Gao, Chen Tang, Xiangyu Yue, Chenyu You

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a giant, cluttered room. Most current "smart" computers (called Multimodal Large Language Models) act like someone who glances at the whole room from the doorway and guesses the answer based on what they think they see. They might say, "I see a bird, so it must have a white belly," without actually checking.

VisReason is a new training program designed to teach these computers to stop guessing and start investigating, just like a human detective would.

Here is how the paper explains it, broken down into simple concepts:

1. The Problem: The "Glance and Guess" Habit

Currently, AI models are great at answering simple questions but often fail when the answer is hidden in a tiny detail or requires understanding how objects are stacked in 3D space. They tend to rely on "shortcuts" (like guessing based on common words) rather than looking closely. It's like trying to read a tiny label on a medicine bottle from across the room; you might guess it says "Aspirin," but you could be wrong.

2. The Solution: A "Zoom-and-Verify" Training Manual

The researchers created a massive dataset called VisReason (and a super-detailed version called VisReason-Pro). Think of this dataset not as a list of questions and answers, but as a step-by-step training manual that forces the AI to show its work.

Instead of just saying "The answer is X," the AI is trained to think out loud in rounds:

Round 1 (The Scan): "I see a bird on a net. I need to look closer at its belly." (The AI draws a box around that area).
Round 2 (The Zoom): Zooms in on the box. "Okay, now I can see clearly. The belly is white and solid."
Round 3 (The Conclusion): "Therefore, the answer is Yes."

The dataset contains 489,000 examples of this process across four different "mystery types":

Text/Documents: Reading tiny text in a receipt or invoice.
Fine-Grained: Distinguishing between very similar things (like different types of birds).
General Questions: Answering standard questions about a scene.
Spatial Relations: Figuring out what is behind or in front of what (e.g., "What is behind the tree?").

3. The Secret Ingredient: "Pseudo-Depth"

One of the unique features of the advanced version (VisReason-Pro) is that it teaches the AI about depth (3D space), even though it only has a flat 2D picture.

The Analogy: Imagine looking at a photo of a street. A human knows the car in the back is "behind" the car in the front. The AI usually struggles with this.
The Fix: The researchers used special tools to estimate how far away objects are (like a "depth map"). They fed this extra info to the AI during training. It's like giving the detective a pair of 3D glasses so they can understand which object is blocking the view of another.

4. The Results: Better Detectives

When the researchers trained their AI models using this new "investigation manual":

They got better at details: They stopped guessing and started finding the specific evidence needed to answer.
They got better at space: They could correctly identify objects hidden behind others or in specific corners.
They became more honest: The AI showed its reasoning steps, making it easier for humans to see why it gave an answer, rather than just a black-box guess.

What the Paper Does Not Claim

It is important to stick to what the paper actually says:

It does not claim this technology is ready for medical diagnosis or self-driving cars yet.
It does not say the AI can now "see" in 3D like a human with eyes; it just learned to use depth clues better than before.
It does not claim the AI is perfect; the paper admits that if the AI zooms in on the wrong spot in the first step, it might get stuck there (a "cascading error").

In a Nutshell

VisReason is a massive library of "show your work" examples that teaches AI models to stop glancing at a picture and start zooming in, checking details, and understanding spatial relationships, just like a careful human would.

Technical Summary: VisReason

Problem Statement

Current Multimodal Large Language Models (MLLMs) have achieved significant progress in visual understanding but remain limited in their reasoning paradigms. Unlike text-only Large Language Models (LLMs), which benefit from Chain-of-Thought (CoT) prompting to articulate intermediate rationales, most MLLMs are optimized for direct input-to-answer mapping. This lack of explicit intermediate supervision leads to several deficiencies:

Shortcut Learning: Models often rely on linguistic priors or global scene statistics rather than performing deliberate, step-by-step visual inspection.
Lack of Spatial Grounding: Existing visual reasoning datasets are often small, domain-specific, or lack structured multi-round supervision necessary for compositional reasoning.
Insufficient Depth Awareness: Most datasets provide only 2D annotations, failing to support reasoning about relative depth, occlusion, and ordinal spatial relations (e.g., "behind," "in front of").

The core problem is the absence of large-scale datasets that capture the human-like, global-to-local reasoning workflow required for complex visual queries, where a model must iteratively zoom into salient regions to verify evidence.

Methodology

Dataset Construction: VisReason and VisReason-Pro

The authors introduce VisReason, a large-scale dataset containing 489,000 annotated examples across four diverse domains: Text/Document Understanding, Fine-Grained Recognition, General VQA, and Spatial-Aware Relation Reasoning.

The dataset is constructed using a unified pipeline that enforces a global-to-local reasoning workflow:

Multi-Round CoT: Each sample includes a multi-turn reasoning trace where the model first assesses the global scene, then identifies a Region of Interest (RoI) via a bounding box, and provides a rationale before potentially zooming in for the next round.
Dynamic Zooming: The pipeline enforces a "zoom-and-verify" behavior. If the target object or text is small or subtle, the model is guided to predict a tighter bounding box and generate a new reasoning step based on the cropped image. This process continues until the answer is derived or a round limit (typically 3–4) is reached.
Depth Augmentation: To address the lack of 3D grounding, the authors integrate pseudo-depth cues derived from monocular depth estimation (DepthAnythingV2) and semantic segmentation (SegmentAnythingV2). These cues provide ordinal depth information (e.g., relative ordering of objects) to guide spatial reasoning.

VisReason-Pro is a high-fidelity subset of 165,000 examples (primarily from the GQA dataset). It is curated using stronger GPT-4.1-series models to generate richer rationales and more precise depth-informed annotations. This subset specifically targets complex spatial relations and small-object queries.

Training Paradigm

The authors propose a training and inference pipeline where the MLLM learns to generate a sequence of actions $Y = (a_0, a_1, \dots, a_T)$ .

Action Definition: Each action $a_t$ consists of a textual reasoning snippet $r_t$ and a predicted bounding box $b_{t+1}$ for the next region of interest.
Iterative Context: The visual context for step $t$ is obtained by cropping the original image $I$ using the bounding box from the previous step. The model autoregressively generates the next token based on the query, the history of actions, and the visual features of the current crop.
Supervision: The models are fine-tuned using Supervised Fine-Tuning (SFT) with a standard negative log-likelihood objective, maximizing the likelihood of the ground-truth reasoning sequence (including both text and serialized box coordinates).

Key Contributions

VisReason Dataset: The release of a large-scale (489K) dataset featuring multi-round, RoI-grounded rationales that explicitly supervise the visual reasoning process rather than just the final answer.
VisReason-Pro: A high-quality, depth-augmented subset (165K) that provides detailed reasoning traces and ordinal-depth supervision, enabling models to reason about spatial relations and occlusion.
Evaluation Suite: A comprehensive benchmark designed to evaluate stepwise reasoning, RoI localization accuracy, and depth-augmented spatial reasoning capabilities.
Unified Pipeline: A demonstration that fine-tuning strong MLLM backbones (e.g., Qwen2.5-VL, InternVL) on this data yields substantial improvements in fine-grained recognition, spatial reasoning, and interpretability.

Experimental Results

The authors evaluated their models (VisReason-7B and VisReason-Pro-7B, based on Qwen2.5-VL-7B) against state-of-the-art MLLMs on the Visual-CoT benchmark and other standard suites.

Performance Gains: VisReason-Pro-7B achieved the highest average score (0.802) among open-source MLLMs, outperforming baselines like LLaVA-NeXT-8B and InternVL-2.5-8B.
Domain-Specific Improvements: The most significant gains were observed in Fine-Grained Recognition (e.g., Birds-200-2011: 0.681 $\to$ 0.831) and Spatial Relation Reasoning (e.g., OpenImages: 0.498 $\to$ 0.805), confirming the efficacy of RoI-grounded supervision.
RoI Localization: On the VisReason-Pro held-out suite, the model achieved superior Intersection over Union (IoU) scores (0.34 at IoU@0.5) compared to baselines, demonstrating improved ability to locate relevant visual evidence.
Depth Reasoning: The model showed a significant improvement in "Grounded Ratio" (0.276 vs. <0.04 for baselines) and reduced depth error, validating the utility of ordinal-depth cues.
Human Evaluation: In a blinded study with 30 raters, VisReason-Pro models scored highest in Grounded Faithfulness (4.46/5) and Stepwise Clarity (4.37/5), indicating more reliable and coherent reasoning chains.
Format Sensitivity: The authors noted that while multi-round CoT excels in spatial and fine-grained tasks, performance on standard short-answer benchmarks (like MME) can drop if the evaluation protocol does not match the verbose CoT format. However, switching to a direct-QA prompt largely recovered performance, suggesting the capability is present but the inference format matters.

Significance and Claims

The paper claims that VisReason addresses the fundamental misalignment between current MLLM training paradigms and human visual cognition. By providing large-scale, process-level supervision that mimics the human "zoom-and-verify" workflow, the dataset enables MLLMs to:

Move beyond shortcut learning and linguistic priors.
Perform systematic, verifiable visual reasoning anchored in localized cues.
Handle complex spatial relations and small-object recognition that are difficult to solve from a single global view.

The authors position VisReason as a cornerstone for cultivating human-like visual reasoning, paving the way for the next generation of multimodal intelligence that can reason through images with the same iterative rigor as text-based reasoning. They acknowledge limitations, such as the reliance on pseudo-depth (not metric 3D) and the risk of cascading localization errors if an initial RoI is misaligned, noting that future work could explore self-correcting rollback mechanisms.