🤖 AI

See2Think: Do Multimodal Models Really Use Intermediate Visual States?

This paper introduces See2Think, a unified evaluation framework comprising a 1,200-sample benchmark and a Visual Action-of-Thought protocol, to reveal that while multimodal models exhibit behavioral dependence on intermediate visual states, their reasoning accuracy is heavily constrained by rendering fidelity and varies significantly across models and environments.

Original authors: Siyu Yan, Zhuoran Yan, Haiying Xu, Panhao Zhou, Jingyu Chen, Chenhao Ji, Shuo Cao, Yongheng Zhang, Haoze Liu, Siyu Zhang, Xiwen Gu, Yihao Liu, Alex Jinpeng Wang

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyu Yan, Zhuoran Yan, Haiying Xu, Panhao Zhou, Jingyu Chen, Chenhao Ji, Shuo Cao, Yongheng Zhang, Haoze Liu, Siyu Zhang, Xiwen Gu, Yihao Liu, Alex Jinpeng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a tricky puzzle, but instead of just looking at the picture, you are allowed to grab a pencil, draw lines, circle clues, and sketch out your thinking on a piece of paper. This is the world of Multimodal Large Language Models—super-smart computer programs that can read text and "see" images. For a while, these models have been getting better at "thinking out loud" by writing down their steps, a trick called Chain-of-Thought. But many puzzles, especially those involving 3D spaces or complex geometry, are hard to solve with words alone. Humans naturally draw diagrams to help us think, so scientists wondered: Can these AI models actually use intermediate drawings and sketches to solve problems, or are they just pretending to?

This is the big question researchers are asking. If an AI says, "I need to draw a line here to figure this out," does it actually use that new drawing to find the answer? Or does it just ignore the drawing and guess the answer anyway? It's like asking if a student is actually using their scratch paper to do the math, or if they are just scribbling nonsense and hoping the final answer comes from magic. Understanding this is crucial because if the models aren't truly using the visual tools they create, we might be wasting time building features that don't actually make them smarter.

Enter See2Think, a new study that acts like a detective for AI reasoning. The researchers built a massive testing ground called See2ThinkBench, containing 1,200 different puzzles ranging from 2D geometry and chemistry diagrams to 3D robot scenes and real-world physics problems. They didn't just ask the models to give an answer; they set up a special protocol called Visual Action-of-Thought (VAoT). Think of this as a game where the AI has to take turns: it thinks, it decides to "draw" something (like highlighting a specific angle or cropping a part of the image), a separate tool actually draws it, and then the AI has to look at that new drawing to continue its reasoning.

The team tested four different powerful AI models to see how they handled this game. They found that there is no "one size fits all" winner. Sometimes, just thinking in words (without drawing) worked best; other times, actually seeing the drawing helped. But here is the twist: the models were surprisingly good at deciding what to draw (picking the right clues), but they often failed at using the drawing correctly. In fact, the study found that the biggest bottleneck wasn't choosing the right action, but rather the "faithfulness" of the drawing itself—did the tool actually draw what the AI asked for?

Perhaps the most fascinating discovery came from a "trick" the researchers played. They intentionally gave the models "corrupted" drawings—pictures that looked like the AI asked for but were actually wrong or misleading. When this happened, the models' performance dropped significantly, especially in 3D scenes, where accuracy fell by over 10 percentage points. This proves that the models were genuinely relying on the visual state they saw, even if that state was broken. However, the study also suggests that just because a model depends on the drawing doesn't mean the drawing actually helped it get the right answer. Sometimes, the models were so dependent on the visual feedback that even a bad drawing confused them enough to fail, showing that "using" a visual state and "benefiting" from it are two very different things.

In short, the paper reveals that while AI models are getting better at the idea of visual reasoning, they still struggle with the messy reality of executing and trusting their own sketches. They are like students who know exactly which part of the diagram to look at, but often get lost when they try to actually read the new sketch they made. The researchers conclude that we need to stop just looking at the final answer and start diagnosing the whole process—checking if the drawing was relevant, if it was drawn correctly, and if the model actually used it to think. Until we fix these bottlenecks, the "thinking with images" might still be more of a performance than a genuine superpower.

Technical Summary: See2Think: Do Multimodal Models Really Use Intermediate Visual States?

1. Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly adopting "visual thinking" strategies, such as generating sketches, annotations, tool calls, and intermediate images to aid reasoning. However, it remains unclear whether these models genuinely rely on these intermediate visual states or if such states are merely decorative. Existing benchmarks suffer from two primary limitations:

Data Limitations: Task collections often lack visual diversity or contain samples solvable via text-only shortcuts (e.g., language priors or detailed captions), failing to isolate the necessity of visual evidence.
Evaluation Limitations: Standard evaluations focus on final answer accuracy or aggregate process quality. They rarely diagnose how intermediate visual states are generated, rendered, or utilized. Specifically, they do not jointly determine if a self-generated visual action is task-relevant, if the rendered state faithfully realizes that action, or if subsequent reasoning behaviorally depends on the returned visual evidence.

2. Methodology: The See2Think Framework

The authors introduce See2Think, a unified evaluation framework designed to disentangle the utility of visual states from behavioral dependence. It comprises two core components:

2.1 See2ThinkBench (The Data Component)

A benchmark containing 1,200 open-ended, visually dependent reasoning problems across 12 task categories, organized into three visual reasoning levels:

2D Structured Reasoning: Geometry, Spatial Puzzles, Physics, Chemistry, Science QA, and Abstract Patterns.
3D Scene Reasoning: Object Attributes and Compositional 3D.
Real-World Visual Reasoning: Robot Manipulation, Robot State Change, Visual Commonsense, and Intuitive Physics.

Construction Principles:

Caption-Only Solvability Filtering: To eliminate text-dominant shortcuts, samples are filtered by testing if a strong multimodal model can answer the question using only a detailed image caption (without the original image). Samples solvable in $\ge$ 3 trials are discarded.
Answer Format Normalization: Multiple-choice options are removed to force free-form generation, except for a small subset (127 samples) where the options are intrinsic to the reasoning problem.
Quality Control: Automatic and manual checks ensure answer uniqueness, image-question consistency, and the necessity of non-trivial visual reasoning.

2.2 Visual Action-of-Thought (VAoT) (The Protocol Component)

VAoT is an inference-time diagnostic protocol that records an interleaved sequence of textual thoughts, structured visual actions, and rendered visual states.

Interaction Loop: Given an image $I$ and question $q$ , the model produces a textual thought $T_t$ and a structured action $A_t$ (e.g., highlight, crop, draw line). An external renderer executes $A_t$ on the current state and returns the updated visual state $R_t$ . The model then reasons based on $H_t = \{T_i, A_i, R_i\}$ .
Controlled Inference Settings: The framework evaluates models under four conditions to isolate specific failure modes:
1. CoT: Textual reasoning only over the original image.
2. VAoT-NoRender: The model plans visual actions but they are not executed/rendered.
3. VAoT (Standard): Actions are executed, and the faithful rendered state is returned.
4. VAoT-WrongRender: Actions are executed, but the renderer returns a task-relevant corrupted state (e.g., a wrong highlight or distorted crop) to probe behavioral dependence.

2.3 Evaluation Metrics

The framework diagnoses the process along three dimensions for a key visual step in each trajectory:

Action Relevance: Does the requested operation target task-relevant evidence?
Render Faithfulness: Does the rendered state correctly realize the requested action?
Feedback Uptake: Does subsequent reasoning use information actually present in the rendered state?

3. Key Contributions

Formulation of Visual-State Use: The paper defines "genuine visual-state use" as a controlled diagnostic problem, distinguishing between the utility of a state and the model's behavioral dependence on it.
See2ThinkBench: A curated benchmark of 1.2K samples with rigorous filtering against text-only shortcuts, covering 12 task categories and three visual reasoning environments.
VAoT Protocol: An observable and intervenable protocol that enables stage-wise diagnosis of action selection, visual execution, and downstream state use.
Comprehensive Diagnostics: The framework provides a unified method to evaluate whether models select relevant operations, render them faithfully, and actually use the resulting evidence.

4. Experimental Results

The authors evaluated four representative models (GPT-5.5, GPT-o3, Gemini 3.5 Flash, and Qwen3-VL-32B-Instruct) across all settings.

4.1 Outcome-Level Findings

No Universal Strategy: No single inference setting dominates across all tasks.
- GPT-5.5 and Qwen3-VL performed best with standard CoT.
- GPT-o3 showed a slight preference for VAoT (closed-loop).
- Gemini 3.5 Flash performed best with VAoT-NoRender (planning without rendering).
Task Dependence:
- 2D Structured: Direct reasoning (CoT) often outperforms visual intervention.
- 3D Scenes: Action planning (VAoT-NoRender) often organizes reasoning better than raw perception.
- Real-World: No single strategy shows a clear aggregate advantage.
Robot Manipulation: Consistently low accuracy across all models, suggesting grounding errors are difficult to compensate for with visual actions.

4.2 Process-Level Diagnosis

Action Relevance: Models consistently select relevant visual operations (scores $\approx$ 0.95–0.99).
Render Faithfulness: This is the primary bottleneck. Scores are significantly lower ( $\approx$ 0.56–0.70) across all models, indicating that models often fail to accurately execute or realize their visual plans.
Feedback Uptake: Models generally use the returned visual information (scores $\approx$ 0.65–0.87), but high uptake does not guarantee accuracy gains.
Failure Modes:
- In 3D scenes, the gap between correct and incorrect trajectories is largest in Render Faithfulness.
- In Real-world tasks, the gap is largest in Action Relevance.
- In 2D tasks, all gaps are small, consistent with the original images already exposing necessary structures.

4.3 Behavioral Dependence (Corrupted Feedback)

Utility vs. Dependence: Correct rendering does not always yield net accuracy gains (e.g., in 2D tasks). However, corrupting the visual state consistently degrades performance, proving behavioral dependence.
Sensitivity: Accuracy drops significantly when feedback is corrupted, particularly in 3D scenes (drops of up to 15.5 percentage points for high feedback uptake).
Rendering Harm: Unfaithful rendering (low Render Faithfulness) often causes "harm" (transitioning from correct to incorrect) in 2D and 3D tasks, whereas faithful rendering mostly reduces disruption.

5. Significance and Claims

The paper claims that See2Think provides the first unified framework to move beyond "visual traces" and final-answer accuracy to evaluate the mechanism of visual reasoning.

Genuine Use: The study demonstrates that while models can plan relevant visual actions, the "genuine use" of intermediate states is often hindered by a failure to faithfully render those actions rather than a failure to plan them.
Conditional Capability: Visual reasoning is not a universal accuracy booster; its effectiveness is highly dependent on the specific model architecture and the visual environment (2D vs. 3D vs. Real-world).
Distinct Concepts: The framework successfully disentangles visual-state utility (does the state help?) from behavioral dependence (does the model rely on the state?). A model may derive little benefit from a correct state but suffer significantly if that state is corrupted, indicating a reliance that does not necessarily translate to improved reasoning quality.

The authors conclude that current multimodal models exhibit a "planning-rending gap," where the ability to select a visual operation is high, but the ability to execute and integrate the result is the limiting factor. The framework offers a controlled environment to diagnose these specific stages of failure.