Technical Summary: OpenVisTool

Problem Statement

Visual tool use has become a critical capability for multimodal agents to acquire fine-grained evidence beyond fixed image encodings. Current prevailing methods for training these agents rely on outcome-based rejection sampling, where teacher-generated trajectories are retained if they are executable and yield a correct final answer.

The authors argue this approach is fundamentally flawed because it conflates outcome validity with causal utility. A strong teacher model may possess sufficient parametric knowledge or reasoning shortcuts to answer a query correctly without actually needing the tool observations it generates. Consequently, imitating such trajectories teaches a student model that tool calls co-occur with correct answers, rather than teaching that tool observations ground the answer. This results in "spurious supervision," where the model learns correlation instead of the causal necessity of acquiring visual evidence.

Methodology: OpenVisTool Framework

To address this, the authors propose OpenVisTool, a three-stage framework designed to synthesize instructive visual tool-use trajectories. The core insight is that a trajectory is only beneficial for learning if it satisfies two joint conditions:

1. Outcome Validity: The trajectory must result in a correct final answer.
2. Causal Utility: The tool observations must causally contribute to the answer; specifically, providing these observations to a fixed "probe" model must improve its answer reliability compared to a no-tool baseline.

The framework operates through the following stages:

1. Difficulty Screening

The pipeline first filters the source pool (covering five domains: Chart, Table, GUI Grounding, Visual Search, and Web-to-HTML) to retain only queries that a capable probe model (Qwen3.5-9B) cannot reliably solve without tools.

• Mechanism: A probe model runs $K$ independent no-tool trials. If the empirical success rate is below a threshold ($\gamma = 0.5$), the query is retained.
• Goal: Ensures the selected problems genuinely demand external visual evidence, avoiding redundant tool calls.

2. Domain-Specific Trajectory Synthesis

For retained queries, a stronger teacher model (Qwen3.5-Plus) synthesizes trajectories using a shared visual toolset.

• Mechanism: Instead of generic prompts, the teacher is guided by domain-specific tool-use patterns ($P_d$) tailored to the visual bottlenecks of each domain (e.g., series isolation for Charts, progressive localization for GUI).
• Goal: Elicits coherent, evidence-acquisition behaviors rather than arbitrary or decorative tool calls.

3. Supervision Verification

The final stage rigorously validates each candidate trajectory against the two-part criterion.

• Outcome Validity Check: Verifies the final answer is correct using domain-specific evaluators.
• Causal Utility Check: The tool calls and observations from the trajectory are fed to the same probe model used in screening (excluding the teacher's reasoning and final answer to prevent leakage). The probe's success rate with tools ($\bar{p}_{tool}$) is compared to its no-tool success rate ($\bar{p}_0$).
• Retention Rule: A trajectory is retained only if it is outcome-valid AND $\bar{p}_{tool} - \bar{p}_0 \ge \delta$ (where $\delta = 0.25$).

Key Contributions

1. OpenVisTool Framework: A novel three-stage pipeline that operationalizes the concept of "instructive supervision" by jointly enforcing outcome validity and causal utility.
2. OpenVisTool-42K: A large-scale, open-source dataset containing 42,000 verified trajectories spanning five visual reasoning domains under a unified toolset.
3. OpenVisTool-Bench: A benchmark constructed to isolate the incremental value of active visual evidence acquisition, focusing on instances where tool access materially improves performance.
4. Empirical Validation: Demonstration that fine-tuning on this dataset improves visual tool-use performance across four backbone models (ranging from 4B to 27B parameters), with results approaching or surpassing leading closed-source systems.

Experimental Results

The authors evaluated the framework using Qwen3.5 variants (4B, 9B, 27B) and Qwen3-VL-8B on OpenVisTool-Bench and out-of-distribution (OOD) benchmarks (Agentic-MME, VTC-Bench).

• Performance Gains: Fine-tuning on OpenVisTool-42K yielded consistent improvements across all models and domains. The average gain on OpenVisTool-Bench was 10.7 points. The most significant improvement was observed in Visual Search (+23.8 points for the 9B model), a domain requiring fine-grained spatial reasoning where tools bridge the gap between perception and task requirements.
• Comparison to SOTA: The 9B model trained with OpenVisTool-42K achieved an average score of 45.8, surpassing GPT-5.5's no-tool performance (41.9) and closely approaching GPT-5.5's tool-augmented performance (49.0).
• Generalization: Models trained on the multi-domain dataset showed consistent gains on OOD benchmarks (Agentic-MME and VTC-Bench), suggesting the learned capability is a transferable meta-skill rather than overfitting to specific tool-calling patterns.
• Ablation Studies:
  • Filter Necessity: Using both outcome validity and causal utility filters outperformed using either filter alone or no-tool trajectory SFT.
  • Cross-Domain Synergy: Training on the full multi-domain mixture resolved strategy conflicts (e.g., between local cropping for search and global rendering for web) better than single-domain training, which often led to performance degradation in incompatible domains.
  • Trajectory Replay: When a probe model was conditioned on tool interactions generated by the instructive-trained model (without seeing the reasoning/answer), it outperformed interactions generated by a model trained only on correctness. This confirms the training induces a more useful evidence-acquisition policy.

Significance

The paper claims that effective visual tool use is not learned merely from successful outcomes but from causally grounded supervision. By shifting the evaluation focus from "does the trajectory end correctly?" to "do the tool calls do real work?", OpenVisTool provides a recipe for constructing training data that teaches models when and how to acquire visual evidence. This approach enables smaller open-source models to leverage visual tools effectively, achieving performance competitive with large closed-source systems, and establishes that the quality of the cold-start data is the primary determinant of a model's ultimate tool-use competence.