Technical Summary: MM-ToolSandBox

Problem Statement

Current benchmarks for evaluating Large Language Model (LLM) agents are predominantly text-centric, failing to capture the complexities of realistic assistant tasks that require visual inputs. While existing frameworks address function calling, stateful interaction, or multi-turn dialogue, they often treat images as static prefixes or limit interactions to single-turn scenarios. Furthermore, many operate within small tool spaces (15–100 tools) and lack the capacity to evaluate agents that must extract information from dynamic visual inputs, map that evidence to specific tool calls, and execute actions within a stateful environment over multiple turns. There is a critical gap in unified frameworks capable of systematically studying visual tool-calling capabilities under dynamic, multi-turn, and large-tool-space settings.

Methodology

To address this gap, the authors introduce MM-ToolSandBox, a unified framework and benchmark designed for evaluating visually grounded tool-calling agents.

Framework Architecture

The framework extends the stateful tool-use environments of ToolSandbox with explicit image handling and visual-specific tools.

• Environment: It simulates 16 application domains (e.g., Shopping, Notes, Files, Music) containing 511 native Python tools.
• Interaction Model: It supports multi-turn, multi-image interactions where images are treated as first-class visual artifacts stored in a shared database, rather than static prompt attachments. Images can be exchanged via tool calls (e.g., send_message_with_image).
• Execution Modes: The framework supports two interfaces:
  1. Code-Execution: Agents write Python code in a sandboxed interpreter to call tools.
  2. Structured Tool-Use: Agents emit schema-constrained tool calls via standard APIs.
• Safety: A three-layer guardrail system (static checks, runtime restrictions, execution limits) ensures safe execution of agent-generated code.

Benchmark Construction

The benchmark comprises 258 human-verified nominal scenarios and 50 interactive UI variants, utilizing 1,284 unique images sourced from eight established vision datasets (e.g., DocVQA, ChartQA-Pro, ScreenSpotPro).

• Design Dimensions: Scenarios are structured along three orthogonal axes:
  1. Information Flow Types: Seven types including aggregate, compare, filter, compute, validate, lookup chain, and cross-reference.
  2. Challenge Types: Cooperative baseline, goal changes, error corrections, and state mutations.
  3. Image Arrival Patterns: Upfront, progressive, late, or mixed arrival of visual inputs.
• Generation Pipeline: An automated six-stage pipeline generates scenarios using information-flow-guided planning, CLIP-based image clustering, and multi-stage quality filtering. This reduces the annotation burden to targeted human review.
• Evaluation Protocol:
  • State Verification: Compares initial and final world-state snapshots against expected entity changes using Entity F1 scores.
  • Rubric-Based LLM Judges: An Agent Judge evaluates five criteria (task completion, instruction following, tool-use validity, no side effects, information accuracy), while a User Judge assesses the coherence of the simulated user.

Key Contributions

1. Unified Framework: A stateful, multi-turn, multi-image simulation environment spanning 511 tools across 16 domains, supporting both code-execution and structured tool-use interfaces.
2. Scalable Benchmark: A dataset of 258 human-verified scenarios and 50 UI variants, generated via an automated pipeline that ensures diverse visual reasoning patterns and conversational dynamics.
3. Systematic Evaluation: A comprehensive evaluation of 12 state-of-the-art models (ranging from 4B open-weight to frontier proprietary systems) revealing distinct failure modes across model scales.

Experimental Results

The authors evaluated 12 models, including GPT-5.4, Gemini 3.1 Pro, Claude 4.5 Opus, and the Qwen 3.5 series.

• Overall Performance: Visual tool calling remains a significant challenge. Even the strongest model, Claude 4.5 Opus, achieved an Agent Success Rate (SR) of only 48.8%. The best-performing model in terms of Entity F1 (GPT-5.4 with high thinking) reached 0.865, but its Agent SR was lower.
• Scaling Laws and Failure Modes:
  • Small Models (<27B parameters): Failures are dominated by planning errors (Task Failure), where models cannot decide the correct sequence of actions.
  • Large Models (≥27B parameters): Failures shift to visual precision errors (Factual Error). For Claude 4.5 Opus, 53% of failures stemmed from incorrect information extraction from images (e.g., misreading text or numbers) despite executing the correct workflow.
  • Multi-Image Memory: Performance degrades significantly as the number of images increases, particularly in "Upfront" scenarios where images are provided at the start, indicating a bottleneck in multi-image working memory.
• Interface Sensitivity: Model performance varies by execution interface. For instance, KIMI 2.6 performed significantly better in Tool-Use mode, while Claude 4.5 Sonnet and Qwen 3.5 preferred Code-Execution.
• UI Mode: In a subset of 50 scenarios requiring interactive UI rendering, performance dropped sharply (e.g., KIMI 2.6 dropped from 45.0% to 26.0% SR), highlighting UI generation as a new, harder frontier.

Significance and Claims

The paper claims that MM-ToolSandBox provides the first unified framework to systematically study visual tool-calling under dynamic, multi-turn, and large-tool-space settings.

The primary significance of the work lies in its failure analysis, which identifies visual precision as the primary bottleneck for capable models, rather than tool-use competence or planning. The authors observe a "planning-to-precision crossover" as model scale increases: smaller models struggle with what to do, while larger models struggle with what they see. This suggests that future research directions must differ based on model capability levels. Additionally, the work highlights that multi-image working memory is a critical limitation for current agents, and that interactive UI generation represents a substantially harder challenge than text-based tool calling.

The authors conclude that visual tool calling is "far from solved," with even frontier models completing fewer than half of the tasks, and emphasize the need for improved visual grounding and reasoning capabilities.