Technical Summary: AsyncTool

Problem Statement

While Large Language Model (LLM)-based agents have demonstrated proficiency in leveraging external tools for complex, multi-step tasks, existing evaluation benchmarks largely overlook the temporal dimension of tool invocation. Specifically, current evaluations typically assume immediate tool responses and are restricted to single-task settings. In real-world applications, however, agents often face tool response latency and must manage multiple concurrent tasks. Existing benchmarks fail to assess an agent's ability to utilize idle waiting periods to advance other independent tasks, a capability the authors term asynchronous tool calling. Furthermore, there is a lack of standardized metrics and protocols for evaluating concurrent tasks with delayed and potentially out-of-order tool feedback.

Methodology

The AsyncTool Benchmark

The authors propose AsyncTool, the first benchmark designed to evaluate LLM-based agents in interactive multi-task environments with delayed tool feedback. The benchmark simulates realistic conditions where:

1. Concurrent Execution: Agents receive multiple heterogeneous tasks simultaneously.
2. Delayed Feedback: Tool calls incur latency; results are not returned immediately.
3. Dependency Tracking: Tasks contain intra-step dependencies, requiring agents to track state and switch between tasks appropriately rather than waiting idly or hallucinating results.

Data Construction

The dataset is constructed via a four-stage pipeline to ensure high quality and diversity:

1. Data Collection: Raw data is extracted from existing benchmarks NESTFUL and BFCLv3, resulting in 12 tools and 358 single-task instances.
2. Coarse Reconstruction: Gemini 2.5 Pro is used to reconstruct task descriptions and generate strictly ordered function-call trajectories aligned with the tools.
3. Fine-grained Annotation: Human annotators verify trajectories for execution validity, dependency compliance, and argument correctness. Errors such as misinterpreted initial conditions or invalid function formats are corrected.
4. Multi-Task Composition: Single-task instances are combined into multi-task scenarios (dual-task and tri-task) using weighted random sampling. The final dataset comprises 712 instances covering various task types (e.g., Data Management, Filesystem, Trading) and dependency structures (SIMILAR and CROSS combinations).

Evaluation Protocol

Performance is assessed at three distinct levels:

• Step Level: Evaluates fundamental tool-calling capability (format, tool selection, parameter correctness) using F1 scores for function and parameter matching.
• Sub-Task Level: Measures trajectory completion and environment consistency (matching hidden states) for individual subtasks.
• Task Level: Evaluates end-to-end success, requiring all subtasks within a task to be completed correctly.

Additionally, the authors introduce efficiency-oriented metrics, specifically the Same-task Streak, to measure how frequently an agent switches between tasks. A lower streak indicates better interleaving ability (utilizing idle time), while a higher streak suggests the agent may be stuck waiting or failing to switch context.

Key Results

The authors evaluated 19 models (including closed-source models like GPT-4.1, GPT-4o, Gemini 2.5 Pro, and various open-source models) on AsyncTool.

• Performance Degradation: Asynchronous execution leads to clear performance degradation compared to synchronous settings. Delayed feedback poses substantial challenges, particularly in maintaining task states and dependencies.
• Model Comparison: GPT-4.1 achieved the highest overall score (38.06), followed closely by DeepSeek-V3.1-Terminus (28.93) among open-source models. Closed-source models generally outperformed open-source counterparts, especially in the Task Level metrics.
• Accuracy vs. Efficiency Trade-off: The analysis reveals a trade-off between task accuracy and scheduling efficiency.
  • High-performing models (e.g., GPT-4.1) balance high accuracy with compact task switching (low Same-task Streak), effectively using idle time.
  • Some models exhibit frequent switching but low accuracy due to dependency violations or tool confusion.
  • Others show high accuracy but poor interleaving (high Same-task Streak), indicating they struggle to utilize waiting periods.
• Failure Modes: Common errors include:
  • Lack of Temporal Reasoning: Prematurely executing the next step in a task before the current tool result returns, often leading to hallucinated parameters.
  • Task Neglect: Focusing on the most recently presented task while ignoring earlier ones.
  • Tool Confusion: Applying tools intended for one task to another.
  • Dependency Violations: Skipping prerequisite steps or mismanaging state.

Significance and Contributions

The paper claims the following contributions:

1. AsyncTool Benchmark: A novel benchmark that jointly considers delayed tool feedback, concurrent multi-task execution, multi-step function calling, and dependency-aware coordination.
2. Diverse Dataset: A high-quality asynchronous multi-tasking dataset constructed via a hybrid data-evolution strategy, covering varied scenarios and dependency structures.
3. Multi-Level Evaluation: A comprehensive protocol assessing performance at step, sub-task, and task levels, coupled with efficiency metrics to analyze task interleaving.
4. Empirical Insights: Extensive experiments identifying that current agents struggle with temporal coordination. The findings highlight that effective asynchronous tool use requires not just frequent switching, but the ability to coordinate switching with dependency tracking and state maintenance.

The authors conclude that temporal coordination is a critical capability for future tool-using agents and that AsyncTool provides a necessary testbed for developing systems with stronger temporal reasoning and efficiency.

Limitations

The paper acknowledges two primary limitations:

1. Dataset Scope: The dataset is reconstructed from NESTFUL and BFCLv3, meaning the range of tools and scenarios is inherently constrained by these sources.
2. Construction Cost: The reliance on high-performance models (Gemini 2.5 Pro) for data reconstruction and the need for extensive human verification make the dataset construction process costly and potentially introduce generation errors that require manual correction.