The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

1. Problem Statement

Current benchmarks for language agents (LLMs) suffer from significant limitations when evaluating their readiness for real-world deployment:

• Lack of Diversity: Existing benchmarks often focus on narrow domains (e.g., only web browsing or only code generation) or single-application tasks.
• Unrealistic Environments: Many benchmarks rely on synthetic, simplified, or empty initial states (e.g., an empty inbox) rather than complex, pre-populated environments found in reality.
• Short Horizons: Tasks are often short, single-step, or require few tool calls, failing to test an agent's ability to manage long-horizon workflows.
• Evaluation Reliability: Many benchmarks use LLM judges or static context checks rather than executing tools and verifying the final state of the environment, leading to unreliable metrics.
• Prompt Fuzziness: Real user instructions are often vague and ambiguous, whereas many benchmarks provide overly detailed, step-by-step prompts that remove the need for agent planning.

2. Methodology: The TOOLATHLON Benchmark

The authors introduce TOOLATHLON, a comprehensive benchmark designed to bridge the gap between simulated agents and real-world execution.

A. Environment and Tools

• Scale: The benchmark spans 32 software applications and 604 tools.
• Diversity: Applications range from daily platforms (Google Calendar, Notion, Gmail) to professional enterprise tools (Snowflake, Kubernetes, BigQuery, WooCommerce).
• Implementation: Most tools are based on Model Context Protocol (MCP) servers. The authors either sourced existing open-source MCP servers or implemented their own, fixing bugs and enhancing functionality to ensure they support the required tasks.
• Realistic State Initialization: Unlike prior works that start with empty states, TOOLATHLON initializes environments with realistic, complex data (e.g., Canvas courses with dozens of students, real financial spreadsheets, email inboxes with hundreds of messages).
  • Hybrid Approach: While remote APIs (like Google Sheets) are used where possible, the authors deploy open-source alternatives locally via containers (e.g., Poste.io for email, WooCommerce for e-commerce) to allow for scalable, reproducible state initialization without needing thousands of real user accounts.

B. Task Design

• Task Volume: 108 manually crafted tasks.
• Complexity: Tasks require interacting with multiple applications, averaging ~20 tool calling turns per task (with a maximum of 100 allowed).
• Fuzzy Prompts: Instructions mimic real user queries—concise, ambiguous, and lacking explicit step-by-step plans. Agents must infer intent from the environment state (e.g., inferring data formats from existing files) and autonomously devise plans.
• Domains: Tasks cover diverse sectors including Education, Finance, Tech, Business, Daily Life, and E-commerce.

C. Evaluation Framework

• Execution-Based Verification: Success is determined by deterministic evaluation scripts that check the final state of the environment (e.g., verifying if a file was created, an email was sent, or a database record updated). This avoids the subjectivity of LLM judges.
• Isolation & Parallelism: Each task runs in an isolated container. The framework supports parallel execution, allowing 108 tasks to be evaluated on a model in ~70 minutes using 10 parallel processes.
• Robustness Handling: The framework includes mechanisms to handle tool errors (allowing the agent to retry rather than crash), overlong tool outputs (truncation and paging), and context management.

3. Key Contributions

1. TOOLATHLON Benchmark: The first benchmark to combine diverse applications, realistic initial states, long-horizon multi-step workflows, and deterministic execution-based evaluation.
2. Realistic State Initialization: A novel approach to simulating complex, pre-populated environments (e.g., full email inboxes, student databases) using local containerized open-source software, addressing the "empty state" limitation of previous benchmarks.
3. Fuzzy Instruction Design: A focus on ambiguous prompts that require agents to infer context and plan, rather than following rigid, pre-defined steps.
4. Comprehensive Evaluation Suite: A lightweight, scalable framework for safe, parallel, and reliable evaluation of agents against real-world constraints.

4. Experimental Results

The authors evaluated state-of-the-art proprietary and open-weight models on TOOLATHLON.

• Overall Performance: Even the best-performing model, Claude-4.5-Sonnet, achieved a Pass@1 success rate of only 38.6% (with an average of 20.2 turns).
• Open-Source Models: The best open-weight model, DeepSeek-V3.2-Exp, achieved a 20.1% success rate, highlighting a significant gap between proprietary and open-source agents.
• Consistency: There is a large gap between Pass@3 (at least one correct trajectory) and Pass^3 (all three trajectories correct), indicating that while models have some capability, they lack reliability and consistency.
• Domain Performance:
  • Claude-4.5-Sonnet: Strongest across most domains, particularly Campus and E-commerce.
  • GPT-5: Performed well in "Daily" tasks but struggled with complex multi-app orchestration.
  • Grok-4: Showed specialized strength in "Tech" tasks.
• Failure Analysis:
  • Tool Calling Errors: Hallucinating non-existent tool names and execution errors (e.g., wrong parameters) are common.
  • Long-Context Challenges: Performance degrades significantly as the number of turns increases. Models often fail to explore all necessary observations or terminate prematurely ("laziness").
  • Overlong Outputs: Models struggle when tool outputs are excessively long (e.g., large HTML or database dumps), often getting trapped in processing them rather than extracting key info.
  • Cost vs. Performance: While Claude-4.5-Sonnet is the most accurate, it is also expensive. Models like DeepSeek-V3.2-Exp offer a cost-effective alternative with moderate performance.

5. Significance and Impact

• Real-World Relevance: TOOLATHLON shifts the focus from "can the model call an API?" to "can the model solve a complex, multi-step business problem in a realistic environment?"
• Diagnostic Tool: The benchmark reveals specific weaknesses in current SOTA models, particularly in long-horizon planning, error recovery, and handling ambiguous instructions.
• Community Resource: The authors have open-sourced the benchmark, the environment setup, and the evaluation framework (toolathlon.xyz), providing a standardized platform for the community to develop more robust, production-ready language agents.
• Future Direction: The results suggest that future agent research must prioritize robust error handling, better long-context management, and the ability to infer intent from fuzzy, real-world prompts rather than just optimizing for single-step tool accuracy.