ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

ThunderAgent is a novel, program-aware agentic inference system that unifies LLM and tool resource management through an "LLM Program" abstraction, achieving significant throughput and memory efficiency gains by optimizing KV cache utilization and enabling asynchronous environment preparation.

The Gist

Imagine you run a busy, high-tech restaurant kitchen.

In this kitchen, you have a team of brilliant chefs (the AI Models) who are trying to solve complex puzzles. Sometimes, the chefs just think and talk (Reasoning). Other times, they need to go to the pantry, the spice rack, or the delivery truck to get ingredients or tools (Tool Execution).

The Problem: The Old Kitchen Chaos

Currently, most AI systems run this kitchen like a chaotic, old-school diner:

1. The "Forgetful" Manager: The manager treats every single order as a totally new, unrelated event. If a chef is in the middle of cooking a complex dish (a "workflow") and has to step away to grab a specific spice (a tool call), the manager immediately throws away all the notes, ingredients, and half-prepared sauces on the counter (the KV Cache) to make room for a new, simple order.
2. The Result: When the chef comes back from the pantry, they have to start the whole dish from scratch. They have to re-chop the onions and re-boil the water. This is called "Thrashing." It wastes massive amounts of time and energy.
3. The Imbalance: Some kitchen stations are packed with chefs working on huge, multi-course meals, while other stations are empty. The manager doesn't realize they can move a chef from the crowded station to the empty one because they don't see the "whole meal" as one connected story.
4. The Clutter: When a chef finishes a dish, the dirty pots and pans (tool environments) are left on the counter. Over time, the kitchen fills up with trash, and there's no room to cook anymore.

The Solution: ThunderAgent

The authors of this paper built ThunderAgent, a new kitchen manager who sees the entire meal as a single "Program" rather than just a series of disconnected orders.

Here is how ThunderAgent fixes the chaos using three simple tricks:

1. The "Program" Passport

Instead of treating every step as a new stranger, ThunderAgent gives every complex task a Passport (called an Agentic Program).

• How it works: This passport tracks the chef's progress, their notes, and their tools. Even if the chef steps away to the pantry, the passport stays with them. The kitchen knows, "Oh, Chef A is just waiting for the oven, but they are still working on Dish #42."
• The Benefit: The manager never throws away the notes while the chef is away. The chef can pick up exactly where they left off instantly.

2. The Smart "Pause" Button

The kitchen has limited counter space (GPU Memory). If the counter gets too full, the old manager would panic and throw away everything.

• ThunderAgent's Trick: It uses a Time-Decay strategy.
  • If a chef is just thinking (Reasoning), they get to stay at the counter.
  • If a chef is waiting for a tool (Acting), they are allowed to stay for a little while. But if they wait too long, ThunderAgent gently asks them to step aside (Pause) to make room for others.
  • Why? It's better to ask a chef to wait for 5 seconds than to throw away 50 minutes of cooking notes. This balances the kitchen perfectly so no one is ever forced to start over.

3. The "Global Waiting Room" & "Clean-Up Crew"

• The Waiting Room: In the old system, if one station was full, new orders had to wait even if another station was empty. ThunderAgent has a Global Waiting Room. If Station A is full, it instantly moves the next chef to Station B. The kitchen is always balanced.
• The Clean-Up Crew: When a chef finishes a dish, the old system left the dirty pots forever. ThunderAgent has a Lifecycle Manager that immediately washes the pots and clears the counter the moment the "Program" is marked as "Done." This prevents the kitchen from ever getting cluttered.

The Results: A Faster, Smoother Kitchen

Because ThunderAgent stops the chefs from re-cooking dishes, balances the work across all stations, and keeps the kitchen clean:

• Speed: The kitchen serves 1.5 to 3.6 times more meals per hour.
• Efficiency: It saves massive amounts of space (memory) and prevents the kitchen from crashing due to clutter.
• Reliability: Even when the pantry is slow or unpredictable (random tool delays), the system adapts and keeps moving.

In a Nutshell

ThunderAgent is like upgrading from a disorganized, forgetful diner to a smart, synchronized restaurant. It understands that a complex task is a journey, not just a series of steps. By keeping the "notes" safe, balancing the workload, and cleaning up instantly, it makes AI agents faster, cheaper, and much more capable of solving real-world problems.

Technical Summary

1. Problem Statement

Current Large Language Model (LLM) inference systems struggle to support complex, multi-turn agentic workflows (e.g., coding agents, scientific discovery agents) efficiently. Existing architectures typically loosely couple a standard LLM inference engine (like vLLM or SGLang) with a general-purpose tool orchestrator (like Kubernetes). This separation leads to three critical inefficiencies:

1. KV Cache Thrashing: Agentic workflows alternate between "reasoning" (LLM generation) and "acting" (tool execution). Existing request-aware systems treat each step as an independent, stateless request. When a tool is called, the system often evicts the KV cache to accommodate new requests. When the tool returns, the system must recompute the entire interaction history (re-prefill), increasing end-to-end latency by up to 7.14× and destroying throughput.
2. Cross-Node Memory Imbalance: In distributed settings, existing schedulers often pin all requests from a single workflow to a specific GPU node to maximize cache locality. However, because agentic workflows have unpredictable context lengths and execution lifetimes, this leads to severe memory imbalance: some nodes run out of memory while others remain underutilized, causing unnecessary preemption.
3. Tool Lifecycle Obliviousness: Orchestrators lack end-to-end visibility into the agent's state. They fail to release resources (e.g., Docker sandboxes, network ports) immediately after a workflow completes, leading to resource exhaustion. Conversely, they often wait for environments to be fully prepared before allowing the LLM to reason, adding significant latency.

2. Methodology: ThunderAgent

ThunderAgent introduces a program-aware abstraction that unifies the scheduling of LLM inference and tool execution. Instead of managing individual requests, it manages Agentic Programs as first-class scheduling units.

A. Program Abstraction

The system models an agentic workflow as a tuple $P = \langle ID, c, T, L, \tau, s \rangle$, where:

• $ID$: Unique global identifier.
• $c$: Context token count (KV cache footprint).
• $T$: Set of required tool environments.
• $L$: Node placement (for spatial locality).
• $\tau$: Execution phase (Reasoning or Acting).
• $s$: Scheduling status (Active, Paused, Terminated).

This abstraction allows the scheduler to distinguish between a temporary tool wait and a terminated session, enabling intelligent resource management.

B. Program-Aware Scheduler

The scheduler optimizes a cost model that minimizes non-productive overheads: recomputation ($Cost_{recompute}$), unused capacity ($Cost_{unused}$), and idle caching ($Cost_{caching}$).

1. State-Aware Pausing & Shortest-First Eviction:
   • The system monitors memory pressure periodically. If a backend is thrashing, it pauses programs.
   • Strategy: It prioritizes pausing programs in the Acting phase (tool execution) to free up memory for Reasoning programs.
   • Eviction Policy: Among programs to be evicted, it selects those with the shortest context lengths first. Theoretical analysis proves that recomputation cost scales quadratically with context length ($c^2$); thus, evicting shorter contexts minimizes the total recomputation penalty.
2. Dynamic Migration via Global Waiting Queue:
   • Instead of pinning a workflow to a specific node, ThunderAgent uses a global program-aware waiting queue shared across all Data Parallel (DP) nodes.
   • When a program is paused, its KV cache is evicted (making it node-agnostic). It can then be restored to any node with available capacity, effectively balancing memory load across the cluster without sacrificing cache locality.

C. Program-Aware Tool Resource Management

1. Lifecycle-Aware Garbage Collection: Resources (Docker containers, ports) are tightly coupled to the program's status. When a program reaches the Terminated state, resources are immediately reclaimed, preventing disk/memory leaks.
2. Asynchronous Environment Preparation: The system overlaps I/O-intensive environment setup (e.g., pulling Docker images) with LLM reasoning. It monitors the global queue and prepares environments for high-priority programs before they are restored to the GPU, hiding initialization latency.

3. Key Contributions

• Unified Abstraction: Introduced the "Agentic Program" concept, decoupling scheduling logic from specific inference backends (vLLM/SGLang) and enabling a unified view of heterogeneous resources (KV cache, disk, network).
• Optimized Scheduling Algorithm: Developed a constrained optimization approach that minimizes recomputation costs via Shortest-First Eviction and balances memory via Global Migration, achieving near-optimal KV cache hit rates.
• Efficient Resource Management: Implemented a lifecycle-aware garbage collector and asynchronous environment preparation, solving the resource leakage and setup latency issues inherent in current agentic systems.
• Open-Source System: The full implementation is open-sourced, facilitating reproducibility and future development.

4. Experimental Results

ThunderAgent was evaluated across diverse workloads (Coding, Routing, Scientific Discovery) and models (GLM-4.6, Qwen-3) on 8×H100 GPU clusters.

• Serving Throughput: Achieved 1.5× to 3.6× throughput improvements compared to state-of-the-art systems (vLLM, Continuum).
  • Example: On SWE-Agent with GLM-4.6, throughput increased by 2.65× over vLLM and 1.52× over Continuum at high concurrency.
• RL Rollout Efficiency: Demonstrated 1.8× to 3.9× improvements in Reinforcement Learning rollout speeds, critical for reducing policy lag in asynchronous training.
• Resource Savings: Reduced disk memory usage by up to 4.2× by effectively reclaiming tool environments.
• KV Cache Hit Rate: Maintained near-100% hit rates for workflows with predictable tool times, whereas baselines dropped to ~60% due to thrashing.
• Latency: Reduced end-to-end latency significantly by eliminating re-prefills and overlapping environment setup.

5. Significance

ThunderAgent represents a paradigm shift from request-level to program-level scheduling in agentic AI.

• Scalability: It solves the "throughput collapse" problem seen in current systems as batch sizes increase, making large-scale agentic inference economically viable.
• Cost Efficiency: By maximizing hardware utilization and minimizing wasted recomputation, it significantly lowers the cost of serving complex agents.
• Foundation for Future Agents: As agents become more autonomous and complex, the ability to manage stateful workflows across heterogeneous resources (GPU, CPU, Network) is essential. ThunderAgent provides the necessary infrastructure to support the next generation of agentic applications.

In summary, ThunderAgent demonstrates that treating agentic workflows as cohesive programs rather than disjointed requests is the key to unlocking high-throughput, low-latency, and resource-efficient agentic inference.