💬 NLP

AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

This paper introduces AGENTSERVESIM, a hardware-aware simulator that accurately models multi-turn LLM agent serving dynamics—including program orchestration, tool-induced gaps, and KV-cache residency—to enable scalable and cost-effective evaluation of serving policies on commodity CPUs without requiring extensive real-system deployments.

Original authors: Rakibul Hasan Rajib, Mengxin Zheng, Qian Lou

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rakibul Hasan Rajib, Mengxin Zheng, Qian Lou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy restaurant.

The Old Way (Standard LLM Serving):
In the past, serving AI models was like a fast-food drive-thru. Every car (request) comes in, orders a burger (a single question), gets it, and drives away. The kitchen doesn't care what the car did before or what it will do next. Each order is independent. If the car comes back later, it's treated as a brand-new customer.

The New Way (Multi-Turn Agent Serving):
Now, imagine the restaurant is hosting a complex cooking competition. A single team (an "Agent") stays at the table for a long time. They ask for an ingredient, then the chef has to wait while the team goes to the pantry to find a spice (a "tool call"), then comes back to ask for the next step. This happens dozens of times in one session.

The Problem: The chef (the AI) has to remember the whole recipe so far. If the team leaves the table to go to the pantry for 5 minutes, the chef shouldn't throw away the notes on the counter (the "KV cache") because they'll need them when the team returns. If the team goes to a different table (a different server) when they come back, the new chef has to re-read the whole recipe from scratch, wasting time.
The Challenge: Managing this is hard. You have to decide: Do we keep the notes on the expensive, fast counter? Do we move them to a slower shelf in the back? Do we throw them away if the team is gone for too long? And which chef should handle the team's next turn?

The Solution: AGENTSERVESIM
The authors built a virtual restaurant simulator called AGENTSERVESIM.

Instead of trying to test these complex rules on real, expensive supercomputers (which costs a fortune and takes forever), they built a digital twin that runs on regular, cheap computers.

Here is how their simulator works, using the restaurant analogy:

The Program Orchestrator (The Head Waiter):
In old simulators, every order was treated separately. This simulator has a Head Waiter who tracks the entire cooking competition as one single "Program." The Waiter knows that Team A is currently waiting for the pantry and won't let the next order start until the pantry trip is done.
The Tool Simulator (The Pantry Timer):
Sometimes the team needs to go to the pantry (run a tool like grep or pytest). These trips can take milliseconds or minutes. The simulator has a special timer that mimics these delays accurately, so the system can test if it's better to keep the recipe notes on the counter or move them to the shelf during the wait.
The Session-Aware Router (The Table Assigner):
If the restaurant has multiple chefs (servers), this router tries to send Team A back to the same chef they started with. This keeps the recipe notes right where they are, saving time. If that chef is too busy, the router calculates the cost of moving the notes to a new chef versus just starting over.
The KV Residency Model (The Note-Taker):
This is the smartest part. It decides where to store the "recipe notes" (KV cache).
- HBM (High-speed counter): Fast but small.
- DRAM/CXL (Back shelf): Slower but bigger.
- The model asks: "Is the team coming back in 10 seconds or 10 minutes?" If it's 10 seconds, keep the notes on the counter. If it's 10 minutes, move them to the shelf so the counter stays free for other teams.

Why is this important?
Testing these strategies on real supercomputers is like trying to test a new restaurant layout by actually building the restaurant, hiring staff, and running it for weeks. It's expensive and slow.

The Result: The authors tested their simulator against real supercomputers (using real AI models and real hardware). They found that the simulator predicts how fast the "cooking competition" will finish with less than 6% error.
The Benefit: Now, engineers can run thousands of "what-if" scenarios on a regular laptop to figure out the best way to manage these AI agents, without needing to rent expensive supercomputers for every single test.

In short: They built a highly accurate "flight simulator" for AI agents that lets researchers practice and optimize how to run complex, multi-step AI tasks without crashing the real plane (or spending a fortune on fuel).

Technical Summary: AGENTSERVESIM

Problem Statement

Modern Large Language Model (LLM) deployments are shifting from stateless, single-turn chat queries to multi-turn agentic systems. Unlike traditional serving, where each prompt-completion pair is an independent request, agentic systems execute stateful programs that interleave LLM calls with external tool invocations (e.g., code execution, web search). These tool invocations introduce variable-duration gaps ranging from milliseconds to minutes.

This shift fundamentally alters serving requirements:

Stateful Execution: User-perceived latency is the Job Completion Time (JCT) of the entire program, not the latency of individual turns.
Cross-Turn Dependencies: Later turns often extend the same conversation, making cross-turn prefix reuse systematic.
KV-Cache Dynamics: KV-cache state must survive tool-induced gaps to avoid costly re-prefilling, yet retaining state during long gaps wastes scarce memory.
Routing Complexity: In multi-instance deployments, routing must preserve session affinity to maintain cache locality, rather than simply load-balancing.

Evaluating scheduling, routing, and KV-cache management policies for these workloads on real hardware is prohibitively expensive. Each design point requires dedicated accelerator time, and the configuration space (arrival rates, model scales, memory hierarchies) is too vast for exhaustive real-world sweeps. Existing simulators (e.g., Vidur, APEX, LLMServingSim) target stateless workloads and fail to capture program identity, tool gaps, or cross-turn KV residency.

Methodology

The authors present AGENTSERVESIM, a hardware-aware simulator designed specifically for multi-turn LLM agent serving. The simulator treats the agent program as the unit of execution rather than individual LLM calls. It is built upon a modular architecture comprising five key components:

Program Orchestrator: Maintains a per-program state machine tracking turn order and outstanding tool states. It ensures that turn $t+1$ is only released after turn $t$ and its associated tool gap have elapsed in simulated time, preserving program identity and dependencies.
Tool Simulator: Materializes tool-induced gaps. It operates in two modes:
- Replay Mode: Uses captured wall-clock durations from real traces for validation.
- Generative Mode: Samples durations from fitted distributions for counterfactual exploration.
Session-Aware Router: Manages multi-instance deployment. It maintains a program-to-instance affinity table to route subsequent turns to the same "home" engine, preserving prefix-cache locality. If the home engine is over-capacity, it calculates the cost of migration or re-prefilling.
Program-Aware Batch Scheduler: Extends the scheduler to key queues on program identity. It exposes program-level state (e.g., attained service) to pluggable policy hooks, allowing for program-level FCFS or attained-service prioritization while retaining standard admission and continuous-batching logic.
KV Residency Model: Tracks KV-cache placement across memory tiers (HBM, Host DRAM, CXL). It assigns policy-defined deadlines to KV blocks based on tool-gap duration and recompute costs. It explicitly models eviction, swapping to host memory, and retention decisions during tool gaps.

The simulator integrates these modules with an operator-graph execution engine (based on LLMServingSim 2.0 and ASTRA-sim) to model heterogeneous hardware, interconnects, and multi-tier memory access.

Key Contributions

Program-Centric Simulation Architecture: A novel framework that abstracts the agent program as the execution unit, enabling the study of turn dependencies, tool-induced idle intervals, and cross-turn KV residency within a single hardware-aware framework.
Validation Against Real Deployments: The authors validated AGENTSERVESIM against real system runs on three hardware platforms (RTX 3090, H100-SXM, B200) using the SWE-Bench Verified workload. The simulator reproduces real-system behavior with less than 6% error across key metrics (JCT and throughput) while running entirely on commodity CPUs.
Hardware-Aware Design-Space Analysis: The paper uses the simulator to systematically study the interaction of prefix caching, scheduling, routing, and KV management. It identifies specific conditions under which agent-aware policies provide benefits, such as the impact of prefix reuse rates and tool latency distributions.

Results

Accuracy: Across 80 configuration cells (varying hardware, model size, and policies), AGENTSERVESIM matched real-system JCT with a mean relative error of <5%. Throughput errors were similarly low (<2%).
Routing Impact: In multi-instance clusters, session-aware routing preserved prefix-cache hit rates near single-instance ceilings (96.26%), whereas round-robin and least-loaded routing dropped hit rates by 3.3–3.6 percentage points, significantly increasing JCT.
Policy Sensitivity: The benefit of agent-aware scheduling (e.g., Autellix, InferCept) over request-level baselines (vLLM-FCFS) is highly dependent on the prefix reuse rate ( $\eta$ ). When $\eta < 0.3$ , recompute costs dominate and policy differences are negligible. When $\eta \ge 0.5$ , choosing the wrong policy can result in a multiplicative increase in JCT.
Tool Latency: As tool latency scales, agent-aware policies (specifically those with adaptive or static-TTL retention) significantly outperform FCFS. For bimodal tool traces, a static threshold retention rule was often sufficient to capture most gains.
Memory Tiering: Adding a second memory tier (CPU DRAM or CXL) mitigates the performance collapse of HBM-only systems under high load. CXL bandwidth provides a measurable benefit (up to ~10% JCT reduction) at specific saturation points, primarily benefiting tail latency.

Significance

The paper claims that AGENTSERVESIM enables controlled, repeatable exploration of agent-serving policies without the need for exhaustive deployment on costly accelerators. By faithfully modeling the stateful nature of agent execution, the simulator allows researchers to:

Evaluate policy interactions (scheduling, routing, KV management) that are tightly coupled and difficult to isolate in real systems.
Explore configurations that are unavailable or too expensive to test physically (e.g., larger clusters, hypothetical memory tiers like CXL, or specific tool-latency distributions).
Move beyond request-level metrics to optimize for program-level objectives like Job Completion Time.

The work establishes a foundation for designing next-generation LLM serving stacks capable of handling the complex, stateful workloads of modern agentic AI.