Technical Summary: Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerf

Problem Statement

The proliferation of Agentic AI applications, which utilize Large Language Models (LLMs) to plan, reason, and execute complex tasks via tool usage, has created a new class of workloads for LLM serving systems. Unlike traditional chatbots that issue independent LLM calls, agentic applications generate dynamic execution graphs where subsequent LLM and tool calls depend on the nondeterministic outputs of previous steps.

Existing benchmarking frameworks fail to address the specific challenges of evaluating serving systems under these conditions:

1. Lack of Reproducibility: Due to the nondeterministic nature of LLM decoding, identical user requests can produce drastically different execution graphs (varying in the number of calls, token counts, and dependencies) across runs. This variance makes it difficult to isolate system performance improvements from workload fluctuations.
2. Inadequate Workload Representation: Current agent evaluation harnesses focus on task completion rather than system metrics (latency, throughput, hardware utilization). Conversely, existing LLM serving benchmarks rely on independent prompts or multi-turn conversations that fail to capture the interdependent, branching nature of agentic workflows.
3. Limited Observability: There is a lack of tools to provide detailed profiling of both the serving system (e.g., KV cache usage, prefix cache hit rates) and underlying hardware (e.g., GPU tensor core utilization) specifically under agentic workloads.

Methodology: XPerf Framework

To address these challenges, the authors present XPerf, an open-source, extensible benchmarking framework designed to load-test LLM serving systems with reproducible agentic workloads. XPerf operates via a trace replay approach consisting of five key stages:

1. Workload Tracing

XPerf employs a generic tracing module to collect execution data from diverse agentic applications without requiring extensive code modifications.

• Instrumentation: It uses library interposition to intercept LLM calls (via standard APIs like OpenAI/Anthropic) and tool calls (via frameworks like LangChain/LangGraph or single-line annotations for custom code).
• Data Collection: It records start/end times, parent-child relationships, and crucially, the exact input and output token IDs for every LLM call. It also traces "orchestrator functions" to understand the nesting of calls.
• Output: All data is logged to a JSONL file, capturing the full execution context.

2. Execution Graph Construction

Raw traces are insufficient for replay because they lack explicit dependency structures. XPerf constructs a Directed Acyclic Graph (DAG) for each user request:

• Time-Span Tree Analysis: It builds a time-span tree from the trace data, where leaf nodes represent LLM/tool calls and internal nodes represent orchestrator functions.
• Dependency Inference: By analyzing the temporal overlap and parent-child relationships within the tree, XPerf infers dependencies between calls. It avoids false dependencies (e.g., between parallel tasks) by respecting the tree hierarchy, ensuring that a call is only scheduled after its specific predecessors complete.

3. Workload Synthesis

XPerf allows users to synthesize new workloads from collected traces to model various serving scenarios:

• Online Serving: Sampling request arrivals from a Poisson process at a target rate.
• Offline Batch Inference: Dispatching a fixed number of requests simultaneously.
• Fixed Concurrency: Maintaining a specific number of in-flight requests.
• Custom Patterns: Users can define specific arrival times via configuration files.

4. Trace Replay

XPerf replays the synthesized workloads against a target serving system with high fidelity:

• Token ID Enforcement: To ensure reproducibility, XPerf submits the exact recorded input token IDs to preserve prefix-cache hit behavior. In a strict mode, it can force the engine to decode the exact recorded output token sequence (useful for MoE models where routing depends on specific tokens).
• Dependency-Aware Scheduling: Calls are issued in the dependency order defined by the constructed DAG. Tool call durations are simulated to maintain realistic timing, ensuring the replayed workload remains faithful even if the target engine differs from the tracing environment.

5. System Profiling

XPerf integrates a lightweight profiler to collect metrics during replay:

• Serving System Metrics: KV cache usage, prefix cache hit rates, token throughput, and queue lengths (via HTTP endpoints).
• Hardware Metrics: GPU tensor core utilization, streaming multiprocessor (SM) usage, and memory bandwidth (via NVIDIA CUPTI).

Key Contributions

1. Reproducible Agentic Benchmarking: XPerf is the first framework to enable reproducible benchmarking of LLM serving systems under agentic workloads by decoupling system performance from the inherent variance of agent execution graphs.
2. Unified Execution Graph Abstraction: It provides a method to automatically construct execution graphs from diverse frameworks (LangChain, LangGraph, custom code) using time-span tree analysis.
3. Detailed Performance Breakdown: The framework offers granular visibility into both software (scheduling, caching) and hardware (GPU utilization) bottlenecks specific to agentic patterns.
4. Open-Source Implementation: XPerf includes eight pre-instrumented agentic applications (covering coding, deep research, Q&A, and general assistance) and is designed for extensibility.

Experimental Results

The authors evaluated XPerf using eight diverse agentic applications and two LLM backbones (gpt-oss-120b and Qwen3.6-35B-A3B) on a dual-H100 GPU testbed.

• Replay Fidelity: XPerf accurately replays workloads with a Mean Absolute Percentage Error (MAPE) in end-to-end request latency of <3% for both isolated and concurrent requests. It successfully reproduces the state of the serving system (running/queued calls) with minimal divergence compared to original runs.
• Performance Characterization:
  • Latency Breakdown: At low load, decoding dominates latency (60–98%). As load increases, queuing delay becomes significant (up to 69% under heavy load).
  • Prefix Cache Thrashing: The study revealed that high request rates can cause "prefix cache thrashing," where the KV cache evicts context needed for subsequent calls in the same agent workflow. This leads to a sharp drop in prefix cache hit rates (e.g., from 91% to <4% in mini-SWE-agent) and increased recomputation costs, negatively impacting throughput.
  • Hardware Utilization: Prefix cache thrashing was correlated with decreased GPU memory bandwidth utilization, even as tensor core utilization remained high, indicating wasted computation.
• Scalability & Routing: In multi-engine deployments, XPerf demonstrated that cache-aware routing policies significantly outperform round-robin routing. Cache-aware routing achieved 3.61× higher throughput on 4 engines compared to single-engine baselines, whereas round-robin only achieved 2.26×, due to the destruction of prefix cache locality.
• Debugging Utility: By minimizing workload variance, XPerf enabled fair comparisons of scheduling policies (e.g., Request-Level FCFS vs. Call-Level FCFS), revealing that observed performance differences in non-reproducible runs were often artifacts of workload variation rather than policy efficacy.
• Overhead: The framework incurs negligible overhead (<0.1% on decoding iteration time, <0.1% on end-to-end latency during tracing).

Significance

The paper claims that XPerf is essential for the rigorous evaluation and optimization of LLM serving systems in the era of Agentic AI. By providing a reproducible, detailed, and extensible benchmarking framework, XPerf allows developers to:

• Identify performance bottlenecks introduced by the dynamic nature of agents (e.g., KV cache thrashing).
• Fairly compare different serving system optimizations and routing policies without the noise of workload variance.
• Understand the trade-offs between concurrency, latency, and throughput in realistic agentic scenarios.
• Debug serving systems by isolating the effects of system changes from the stochastic behavior of agents.

The authors emphasize that while the framework focuses on the serving system, the ability to faithfully replay execution graphs lays the necessary foundation for future work that may extend benchmarking to the host-side application logic and tool execution environments.