Building Interactive Real-Time Agents with Asynchronous I/O and Speculative Tool Calling

`) followed by specific actions: calling a tool, pausing to wait for more information, or providing a final answer. It can also handle interruptions where the context is updated mid-generation.

2. Speculative Tool Calling

To minimize latency, the system aims to launch tool calls as soon as sufficient information is available, even if the user's input is incomplete.

• Handling Uncertainty: The system manages the risk of incorrect tool calls by categorizing tools as either "safe" (read-only) or "unsafe" (state-modifying).
  • Safe Tools: Can be executed speculatively based on partial information.
  • Unsafe Tools: Execution is held up until a "commit point" is reached (defined as receiving the final user query update and the model indicating no further plan modifications are needed).
• Correction Mechanism: The framework allows the model to modify or remove tool calls that have been issued but not yet executed. If a tool call is invalidated by new information, the system can cancel the pending call (injecting a cancellation message) or edit the arguments. If a speculative read-only tool has already executed but is later found to be incorrect, the system discards the observation and prevents downstream dependencies from using it.

3. Training and Inference Framework

• Clock-Based Training: To adapt models to this asynchronous behavior, the authors introduce a clock-based training methodology. The training environment simulates delays by converting time intervals into token counts based on a tokens-per-second rate. Events (user updates, tool results) are buffered in an event queue and injected into the model's context at specific "clock" ticks, forcing the model to learn to reason and act while waiting for information.
• Synthetic Data Generation: The authors synthesize training trajectories using a strong teacher model (Qwen-2.5-32B-Instruct). They align ground-truth tool calls with the earliest possible moment they could be called based on streaming input segments. Crucially, they generate examples where the model must correct erroneous early tool calls, teaching the agent to handle the "edit and remove" logic required for speculative execution.
• System Architecture: The inference system features a core event-driven agent loop (managing context and reasoning) and a decoupled task manager (handling tool dependencies, parallel execution, and speculative safety checks).

Key Contributions

1. Asynchronous I/O: A method to decouple agent reasoning from user and environment delays, enabling continuous processing while waiting for external inputs.
2. Speculative Tool Calling: A strategy to launch tool calls early, with specific protocols for handling read-only vs. state-modifying tools, and mechanisms to modify or cancel in-flight tool calls based on updated information.
3. Clock-Based Training & Synthetic Data: A novel training pipeline that simulates streaming delays and generates data for Supervised Fine-Tuning (SFT), including examples of error correction for speculative tool calls.
4. Unified Framework: An event-driven inference system that integrates asynchronous user inputs, tool execution, and speculative correction into a single cohesive loop.

Results

The authors evaluated their approach on both closed-source cloud APIs (OpenAI Realtime API) and open-source edge-scale models (Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct) across two benchmarks: HotpotQA (multi-hop search) and TinyAgent (multi-step personal assistant workflows).

• Cloud Models: Applied to the OpenAI Realtime API, the method achieved 1.3× to 1.7× speedups in latency with only minor accuracy loss compared to a standard synchronous reason-and-act baseline.
• Edge Models: For the 3B parameter models, the approach yielded 1.6× to 2.2× speedups over a strong SFT baseline. The models trained with the proposed "Async-SFT" methodology maintained accuracy comparable to non-streaming SFT baselines while significantly reducing latency.
• Naturalistic Evaluation: On a custom dataset of human-produced naturalistic utterances (containing fillers, hesitations, and mid-utterance corrections), the streaming models showed lower accuracy than on synthetic benchmarks, highlighting challenges in handling real-world speech disfluencies and intent drift, though the framework successfully demonstrated the ability to process these inputs.

Significance and Claims

The paper claims to provide a generalizable solution for enabling real-time, low-latency interaction in agentic AI systems. By decoupling the agent's processing from external delays and introducing a robust mechanism for speculative execution and correction, the work demonstrates that complex multi-turn tool workflows can be performed with latency suitable for voice interactions (under 1 second). The authors emphasize that their approach allows existing cloud APIs to be used out-of-the-box with speedups, while also providing a training methodology to adapt smaller, edge-scale models for real-time streaming scenarios. The work positions itself as a unifying step forward, addressing the gap between asynchronous user inputs and asynchronous environment responses, which previous works had not tackled simultaneously.