Technical Summary: OmniaBench

Problem Statement

Large Language Models (LLMs) are rapidly evolving from static text generators into general agents capable of understanding user intents, invoking external tools, and completing complex tasks through interaction. However, existing benchmarks for evaluating these agents suffer from significant limitations: they often focus on narrow scenarios, restricted tool ecosystems, or isolated interaction formats. This lack of diversity makes it difficult to systematically characterize model capabilities across the heterogeneous application settings found in the real world. There is a critical gap between current benchmark coverage and the diverse, multi-turn, stateful, and constraint-heavy nature of real-world agent use cases.

Methodology

1. Taxonomy and Scenario Construction

OmniaBench addresses this gap by constructing a hierarchical taxonomy grounded in real-world application ecosystems. The authors derive scenario knowledge from app stores, product requirement documents (PRDs), industry resources, web retrieval, and human refinement. This results in a taxonomy spanning three primary settings:

• ToC (Consumer): 22 Level-1 and 101 Level-2 domains.
• ToB (Business): 38 Level-1 and 186 Level-2 domains.
• ToE (Employee): 30 Level-1 and 67 Level-2 domains.
  The total scope covers 90 Level-1 and 354 Level-2 domains, providing a significantly broader domain space than benchmarks centered on single tool ecosystems.

2. Environment and Task Synthesis

For each domain, the team constructs executable environments containing entities, state variables, tools, and initialization configurations. These are instantiated as Python classes with controlled sandbox environments for file and code operations. Tasks are synthesized through four complementary routes to ensure diverse interaction forms:

• DAG (Directed Acyclic Graph): Focuses on multi-turn stateful interaction and tool-chain execution.
• DAG-S: Derives single-turn tasks from DAG-based tasks through query refinement.
• Solver: Targets selection, scheduling, allocation, and optimization scenarios using implicit or explicit solver-guided synthesis.
• Program: Focuses on complex procedural reasoning involving branching, iteration, task-program synthesis, and debugging.

The resulting dataset contains 1,431 tasks, with a curated "challenging set" of 644 tasks designed to reduce evaluation costs and mitigate data contamination.

3. Evaluation Framework

OmniaBench employs a unified trajectory-based evaluation framework within a Partially Observable Markov Decision Process (POMDP) formulation.

• Metrics: The primary metric is Pass@1.
• Scoring: Tasks are evaluated using a ten-dimensional capability taxonomy (Task Understanding, Information Gathering, Planning & Decision Making, State Management, Tool Use, Code & Programmatic Operations, Data Analysis, Office & Document Handling, Interactive Collaboration, Reliability & Safety).
• Verification: Rubric-based criteria are used for general tasks, while VerifyCode is employed for program-based tasks to determine binary pass/fail outcomes based on trajectories and final observations.
• Simulation: Multi-turn interactions utilize persona-grounded user simulators to control difficulty via preferences, communication styles, and information disclosure.

Key Contributions

The paper outlines four primary contributions:

1. OmniaBench Benchmark: A scenario-rich, challenging benchmark integrating real-world app stores, PRDs, and web search to construct a broad taxonomy (ToC/ToB/ToE) instantiated into executable, evaluable agent tasks.
2. Multi-Dimensional Capability Taxonomy: A framework characterizing model capabilities across ten distinct dimensions, enabling fine-grained diagnostic analysis beyond aggregate success rates.
3. Compositional Atomic Difficulty Framework: A design organizing task challenges around user intent, persona constraints, environment states, tool execution, multi-turn interaction, error recovery, and result verification.
4. Systematic Evaluation: A comprehensive evaluation of both closed-source and open-source models, providing evidence on capability boundaries, domain specialization, and error patterns.

Results

The benchmark presents substantial challenges to current frontier models:

• Performance Ceiling: Even the most advanced models tested, Claude-Sonnet-5 and GPT-5.6-Sol, achieved Overall Pass@1 scores of only 58.54% and 57.14%, respectively.
• Capability Variance: Performance is highly non-uniform across the ten capability dimensions. Models with similar overall scores exhibit markedly different strengths (e.g., in Task Understanding vs. State Management).
• Domain Variance: Significant performance differences exist across ToB, ToC, and ToE splits. For instance, while some models excel in ToB, their performance may drop in ToE, indicating that a single aggregate metric cannot fully characterize practical applicability.
• Efficiency: Models with comparable success rates often differ significantly in tool-use efficiency. Some require redundant exploration and circuitous interactions, while others achieve outcomes with compact, targeted tool sequences.
• Error Analysis: Reasoning-related failures account for 53.8% of errors, with planning/decomposition (36.7%) and constraint violations (16.5%) being major contributors. Direct tool-use errors (e.g., formatting) constitute a smaller proportion, suggesting bottlenecks lie in long-horizon planning and adaptive correction rather than basic invocation.

Significance

OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents. The authors argue that as overall task success rates improve, a single aggregate score becomes insufficient. Instead, the benchmark offers a systematic foundation for analyzing specific abilities, domain specializations, and failure patterns. By revealing persistent limitations in planning, constraint maintenance, and adaptive correction, OmniaBench serves as a critical tool for guiding the development of more robust, efficient, and reliable general AI agents capable of operating in diverse, real-world scenarios.