Technical Summary: AsyncFC – Concurrency without Model Changes

Problem Statement

Modern Large Language Model (LLM) agents rely on function calling (tool use) to interact with external APIs and perform actions. However, the prevailing execution semantics are strictly synchronous: the model's decoding process is blocked until a function call completes and its result is appended to the conversation history. This serialization creates a significant bottleneck, increasing end-to-end latency, particularly in workflows involving multiple sequential or parallel tool calls.

Existing approaches to mitigate this often require altering the function-calling protocol, modifying the model architecture, or shifting concurrency management complexity onto the model itself. These changes can compromise the standard call–return semantics, introduce race conditions, or necessitate additional fine-tuning.

Methodology: AsyncFC

The authors introduce AsyncFC, a pure execution-layer framework that decouples LLM decoding from function execution without requiring any changes to the underlying models, function implementations, or the standard synchronous function-calling protocol.

Core Mechanisms

1. Future-Based Decoding:

   • AsyncFC transforms synchronous function schemas to support future-valued inputs and outputs.
   • When the model emits a function call, the runtime dispatches the execution to a backend executor and immediately returns a symbolic future placeholder (an unresolved result) to the model.
   • This allows the model to continue decoding subsequent turns without waiting for the function to complete, satisfying the standard call–return protocol while enabling decode–execution overlap.
   • The framework automatically transforms function inputs to accept either concrete values or future identifiers, allowing the model to reason over unresolved results as arguments for subsequent calls.

2. Event-Driven Result Integration:

   • Once a function execution completes, AsyncFC proactively integrates the resolved result into the model's context at turn boundaries by inserting user messages that bind the future identifier to its value.
   • If the model explicitly requires a concrete value (e.g., via an await_future call), the runtime detects this intent early to minimize latency penalties, potentially utilizing parallel decoding streams to reduce overhead.

3. Dependency-Aware Scheduling:

   • To safely execute functions in parallel, AsyncFC employs a scheduler that manages concurrency based on resource dependencies.
   • Default Behavior: Without annotations, the scheduler conservatively serializes execution to ensure correctness.
   • Optimized Behavior: Developers can provide lightweight dependency annotations (via decorators) specifying read/write sets for function resources. The scheduler uses these to derive a Task-specific Function Dependency DAG, permitting concurrent execution only when dependencies are satisfied (i.e., no conflicting state access).
   • This approach unlocks inter-function parallelism while maintaining deterministic state updates and avoiding race conditions.

4. Error Handling:

   • If a function fails, AsyncFC cancels all transitively dependent subsequent calls while leaving independent calls untouched. Failure messages are injected into the context to trigger recovery turns.

Key Contributions

• Protocol-Compatible Asynchrony: AsyncFC is the first framework to enable asynchronous function calling and inter-function parallelism while strictly preserving the standard synchronous call–return protocol expected by off-the-shelf LLMs.
• Zero Model Modification: The system requires no fine-tuning, architectural changes, or modifications to existing function implementations. It operates entirely at the execution layer.
• Native Reasoning Discovery: The work demonstrates that LLMs possess a native capability to reason over symbolic future placeholders representing unresolved execution results, enabling an asynchronous paradigm for model-tool interaction without explicit training.
• Generalizability: The framework extends beyond external tool use to asynchronous thinking, where sub-tasks are treated as long-latency function calls, allowing the main model to coordinate parallel reasoning agents.

Experimental Results

The authors evaluated AsyncFC across standard function-calling benchmarks (BFCL v3/v4), software engineering tasks (SWE-bench Lite), and multi-hop reasoning (HotpotQA).

• Latency Reduction:
  • BFCL v4 Web Search: Achieved a 1.26× speedup over Sequential FC and a 1.12× speedup over Parallel FC under realistic function latencies.
  • SWE-bench Lite: Integrated with SWE-agent, AsyncFC achieved a 1.44× speedup over the baseline SWE-agent and a statistically significant improvement over a Parallel FC variant.
  • HotpotQA: Demonstrated a 1.24× speedup on multi-hop reasoning tasks.
• Accuracy Preservation: Across all benchmarks, AsyncFC showed no statistically significant degradation in task accuracy compared to synchronous baselines. This confirms that models can successfully reason over unresolved futures.
• Scalability: Latency savings were shown to scale with function execution time. Initially driven by decode–execution overlap, gains are increasingly dominated by inter-function parallelism as function latencies increase.
• Cross-Model Robustness: The framework successfully improved performance on different models (GPT-4o, Gemini 3.1 Pro, GPT-5.2) and in scenarios with and without explicit dependency annotations (though annotations yield higher speedups).

Significance and Claims

The paper claims that AsyncFC addresses a critical bottleneck in agentic workflows by offloading concurrency management from the model to the system runtime. Its significance lies in three areas:

1. Efficiency: It provides a practical, drop-in solution to reduce end-to-end latency in tool-using agents, particularly for long-latency operations, by maximizing exploitable parallelism without compromising the standard interaction protocol.
2. Capability Insight: The results provide empirical evidence that modern LLMs can natively reason over symbolic futures. This suggests that the "asynchronous" capability is an emergent property of models trained on software artifacts containing asynchronous patterns (e.g., promises/futures), rather than a feature requiring explicit architectural intervention.
3. System Design: By introducing explicit execution-order semantics and dependency-aware scheduling, AsyncFC resolves ambiguities present in existing parallel function-calling APIs, reducing failure modes caused by race conditions while enabling safe, dependency-safe parallelism.

The authors conclude that this asynchronous model–tool interaction pattern offers a new paradigm for designing flexible and efficient agentic workflows, applicable to both external tool use and internal reasoning processes.