Technical Summary: Agent-X

Problem Statement

Deep reasoning is essential for solving complex, real-world tasks that require sequential, multimodal understanding. However, existing benchmarks for agentic systems suffer from significant limitations: they predominantly rely on fully synthetic, single-turn queries; they lack support for diverse visual modalities (such as video and multi-image comparisons); and they fail to provide frameworks for assessing the quality of reasoning over multiple steps. Current evaluations often focus solely on final answer accuracy, neglecting the interpretability, logical coherence, and tool-use fidelity required for genuine agentic behavior in authentic settings.

Methodology

To address these gaps, the authors introduce Agent-X, a large-scale benchmark designed to evaluate vision-centric agents in real-world, multimodal environments.

Benchmark Construction

Agent-X comprises 828 agentic tasks derived from authentic visual contexts, including images, videos, and instructional text. The construction follows a semi-automated pipeline:

1. Query Generation: Large Multimodal Models (LMMs) generate initial candidate queries based on visual inputs and a predefined toolset.
2. Human Refinement: Human annotators refine these queries to ensure realism, clarity, and the necessity of multi-step reasoning. Crucially, queries are designed without explicit references to specific tools or step sequences, forcing agents to plan independently.
3. Reasoning Trace Creation: LMMs generate initial reasoning traces (step-by-step tool calls, inputs, outputs, and thoughts). Human reviewers then rigorously verify and correct these traces for logical consistency, factual accuracy, and proper tool usage.
4. Ground Truth: Each task includes a ground-truth reasoning trace, a final answer, and a justification.

Task Scope and Tooling

The benchmark spans six major agentic environments:

• General Visual Reasoning
• Web Browsing
• Security and Surveillance
• Autonomous Driving
• Sports
• Math Reasoning

Tasks require the integration of 14 executable tools (e.g., OCR, SceneDescriber, ObjectCounter, Calculator, WebSearch, CodePlotter) across perception, visual operation, math, and artistic domains. The dataset includes 716 image inputs and 112 video inputs.

Evaluation Framework

Agent-X introduces a fine-grained, step-level evaluation framework that assesses agents across three modes:

1. Step-by-Step: Evaluates intermediate reasoning quality using metrics like Grounding Score ($G_s$), Tool Precision ($T_p$), and Tool Accuracy ($T_a$).
2. Deep Reasoning: Assesses the coherence and logical consistency of the entire reasoning chain using Faithfulness ($F_{acc}$), Context Score ($C_s$), Factual Precision ($F_p$), and Semantic Accuracy ($S_{acc}$).
3. Outcome: Measures the correctness of the final answer (Goal Accuracy, $G_{acc}$) and the overall correctness of the toolset used (Toolset Accuracy, $T_{sacc}$).

The evaluation employs multiple judges (GPT-4o, Qwen-14B, and human annotators) to ensure robustness and minimize bias.

Key Contributions

1. Agent-X Benchmark: The first large-scale benchmark combining real-world multimodal inputs (images, video, text) with tool-augmented, step-wise reasoning evaluation across six diverse environments.
2. Fine-Grained Metrics: A novel suite of metrics that evaluates not just the final outcome but the intermediate reasoning steps, tool selection accuracy, and logical coherence, distinguishing between genuine reasoning and confabulation.
3. Comprehensive Evaluation: An extensive evaluation of 12 state-of-the-art Large Multimodal Models (LMMs), including closed-source models (GPT-4o, Gemini-2.5-Pro, OpenAI-o4-mini) and open-source models (Qwen2.5-VL, InternVL, VideoLLaMA3).

Results

The evaluation reveals significant challenges for current models in vision-centric agentic tasks:

• Low Full-Chain Success: Even the best-performing models achieve less than 50% Goal Accuracy ($G_{acc}$). The top performer, OpenAI-o4-mini, reaches 45%, while most open-source models score below 30%.
• Reasoning vs. Execution: Models with strong reasoning capabilities (high $F_{acc}$, $F_p$) tend to perform better on final outcomes, but tool invocation and argument prediction remain core bottlenecks.
• Tool Use Limitations: Tool-related metrics show wide variation. Models frequently hallucinate tool usage, produce invalid JSON formats, or fail to chain tools correctly. For instance, Gemini-1.5-Pro exhibits high JSON error rates (44.5%), while GPT-4o struggles with visual misinterpretation (18.5%).
• Open-Source Progress: While closed-source models currently lead, top open-source models like Qwen2.5-VL-7B show competitive performance in specific metrics, indicating a narrowing gap.

Significance

The paper claims that Agent-X provides a critical step forward in evaluating agentic systems by shifting focus from surface-level answer correctness to deep, interpretable reasoning. The findings highlight that current LMMs struggle with the complex interplay of perception, planning, and tool execution required for real-world tasks. By identifying specific bottlenecks—such as temporal reasoning in videos, tool schema adherence, and structured output enforcement—the benchmark offers actionable insights to guide future research in developing more robust, reasoning-centric multimodal agents. The authors posit that Agent-X will drive advancements in multimodal reasoning and robust tool integration, serving as a standard for tracking progress in vision-centric agentic frameworks.