The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

Technical Summary: The Illusion of Visual Tool-Use

Problem Statement
The "thinking-with-images" paradigm equips Multimodal Large Language Models (MLLMs) with active visual operations, such as crop-and-zoom, to gather fine-grained evidence beyond a single holistic view. However, empirical observations reveal a "marginal improvement paradox": models utilizing these tools often achieve only marginal or even negative accuracy gains compared to direct inference, despite incurring substantially higher token costs. Furthermore, tool-augmented rollouts frequently fail on questions that direct inference answers correctly, often by repeatedly cropping irrelevant regions. The central question addressed is whether the visual evidence returned by these tools causally affects the model's final prediction, or if the tool-use is merely a performative ritual.

Methodology: A Causal Audit Framework
The authors formulate visual tool-use as a Structural Causal Model (SCM) over the input image $I$, query $Q$, a sequence of tool actions $T_{1:n}$, returned observations $O_{1:n}$, and the final answer $Y$. This framework distinguishes between two causal paths from input to output:

1. Observation-mediated paths ($T \to O \to Y$): Where the visual content of the observation genuinely grounds the answer.
2. Action-induced shortcuts ($T \to Y$): Where the mere presence of a tool action influences the answer, regardless of the observation's content.

To audit these paths, the paper proposes a three-level intervention protocol:

• Policy-Level: Compares tool-enabled policies ($\pi_{tool}$) against direct inference policies ($\pi_{direct}$) to measure the Average Treatment Effect (ATE) on accuracy.
• Trajectory-Level: Dynamically corrupts all returned observations ($O_i$) during rollout (e.g., replacing them with random crops from the same image) to measure the total effect of removing visual evidence, including induced policy reactions like budget exhaustion.
• Step-Level: Introduces Visual Evidence Gain (VEG), a counterfactual estimand. For a specific step $i$, the model's confidence gap is measured before the call, after the real observation, and after a counterfactual observation (a crop from a different location). $VEG_i = \Delta M^{real}_i - \Delta M^{cf}_i$ isolates the causal contribution of the specific visual content, filtering out the action-induced shortcut.

Key Contributions

1. Causal Formulation: The paper operationalizes visual tool-use as a causal graph, disentangling observation-mediated utility from action-induced shortcuts via a three-level intervention protocol.
2. Identification of Policy Miscalibration: The study identifies "policy miscalibration" as the central bottleneck, characterized by two distinct failure modes:
   • Calling Without Looking (CWL): The model calls tools, but the visual content has no causal effect on the answer (either due to saturated prior confidence or structurally inactive calls).
   • Looking Without Planning (LWP): The visual content is informative, but the calling schedule is incoherent (e.g., continuing to call after the answer is already determined or exhausting the budget on uninformative regions).
3. Trajectory-Level Diagnostic: A deterministic classifier sorts every rollout into four groups: No-call, Mode 1 (CWL), Mode 2 (LWP), and Calibrated (effective use).
4. Decomposition of Gains: The authors decompose the aggregate policy-level accuracy gain into contributions from these groups, revealing that improvements are concentrated almost entirely in the Calibrated minority.

Results
Evaluated across six representative models (including DeepEyes, Mini-o3, Qwen3-VL, and Thyme) and five fine-grained perception benchmarks:

• Marginal Gains: Policy-level ATEs are often small or negative. For instance, DeepEyes shows essentially zero improvement, while Mini-o3 and Qwen3-VL-8B show modest gains on specific benchmarks.
• Causal Ineffectiveness: Trajectory-level interventions (corrupting observations) cause massive accuracy drops in models like Mini-o3 and Qwen3-VL, suggesting they rely heavily on the tool interface structure rather than the visual content, or that their planning collapses when feedback is corrupted.
• VEG Analysis: Step-level analysis reveals that for many models (e.g., DeepEyes), VEG is near zero across all calls, indicating the visual content is ignored. For others (e.g., Qwen3-VL-8B), VEG is high only when the model is not yet saturated, but the model frequently continues calling after saturation (Over-Extension).
• The Illusion: The Calibrated subset (where tools are used effectively) constitutes a minority of rollouts (e.g., ~20-45% depending on the model). The aggregate accuracy gains reported in benchmarks are driven almost exclusively by this subset, while the majority of tool calls are either decoupled from the answer (CWL) or scheduled incoherently (LWP).

Significance and Claims
The paper claims to expose the "illusion of visual tool-use": the aggregate accuracy gains observed in benchmarks create a misleading impression that visual tool-use is broadly and causally effective. In reality, for a broad range of rollouts, the returned visual evidence does not causally mediate the answer.

The authors propose an RL-trap hypothesis, suggesting that outcome-only reinforcement learning (rewarding only the final correct answer) reinforces the $T \to Y$ shortcut (Mode 1) and fails to penalize redundant or harmful intermediate steps (Mode 2), as both a Calibrated and a Mode 1 trajectory receiving the same final answer get identical rewards.

The significance of this work lies in shifting the evaluation focus from "does the model call tools?" to "does the model think with images?" The proposed diagnostic offers operational uses, such as beyond-accuracy evaluation metrics, inference-time adaptation (bypassing tools for Mode 1 or early-stopping for Mode 2), and process-aware credit assignment to improve training. The authors note that their conclusions are limited to open-source models with white-box access for step-level interventions and focus primarily on the crop-and-zoom operation.