Technical Summary: TraceLab: Characterizing Coding Agent Workloads for LLM Serving

Problem Statement

While agentic Large Language Models (LLMs), particularly coding agents like Claude Code and Codex, are rapidly becoming major applications, efficiently serving them remains a significant challenge. Existing public traces and benchmarks fail to capture the specific characteristics of real-world, day-to-day coding-agent usage required for serving-system analysis.

• Existing Traces: Systems like Mooncake and Splitwise capture real user interactions but focus on traditional chat or single-turn completion, lacking the long-horizon, multi-step, tool-calling structure of coding agents.
• Existing Benchmarks: Benchmarks like Terminal-Bench and SWE-bench evaluate model accuracy on isolated tasks. They do not capture the diversity of real-world usage, such as persistent sessions, accumulated context, and human-paced gaps between requests, which are critical for understanding serving costs and system behavior.

Consequently, there is a lack of data to guide the optimization of LLM serving systems for the unique workload patterns of coding agents.

Methodology

To address this gap, the authors collected, normalized, and analyzed a large-scale, cross-provider trace of real coding-agent usage.

• Data Collection: The dataset comprises approximately 4,300 sessions collected over eight months from 43 developers. It includes roughly 350,000 LLM steps and 430,000 tool calls.
• Sources: Data was gathered from the daily research workflows of the authors using Claude Code (Anthropic) and Codex (OpenAI), spanning over 20 model versions.
• Processing Pipeline: The authors built a pipeline to extract raw logs, normalize event structures and token accounting (decomposing inputs into prefix tokens for cached history and append tokens for new input), and anonymize the data by replacing identifiers and removing raw text while retaining token counts and timestamps.
• Analysis Scope: The study analyzes macroscopic metrics (session duration, cost, context growth), LLM generation characteristics (input/output lengths, timing), tool-calling behavior (latency, distribution), and prefix cache efficiency (hit rates, eviction policies).

Key Findings

1. Session and Context Characteristics

• Autonomous Loops: Coding sessions are largely autonomous. A typical session involves an average of 8.8 LLM calls and 10.8 tool calls per request. Most steps are tool-initiated rather than user-initiated.
• Context Growth: Contexts generally grow as new inputs and tool results are appended. However, context compaction (summarizing history to stay within limits) occurs in about 9.7% of sessions, primarily triggered by tool-initiated steps near the context limit.
• Cost Structure: Despite prefix tokens being billed at roughly 1/10th the rate of fresh input, they dominate the cost (59.5% of total spend) because the accumulated context is replayed on every step. Output tokens account for only 11.2% of costs.
• Time Distribution: Sessions are heavily dominated by human "thinking" time (92.3% of wall-clock time). Within a request, tool execution (59.8%) and LLM generation (41.0%) contribute comparably to response time.

2. LLM Generation Patterns

• Long Context, Short Output: A typical step reads a massive context (median ~119k prefix tokens) but generates a short output (median ~214 tokens). This contrasts with traditional reasoning workloads.
• Bimodal Input Structure: There is a distinct split in input patterns. Steps with short prefixes (often cache misses or initial requests) have large append tokens, while steps with long prefixes have very small, incremental append tokens.
• Output Attribution: Providers differ in how they handle previous outputs. Claude and GPT-5.5 often re-send previous outputs as new append tokens, whereas GPT-5.4 often caches them as part of the prefix.
• Decoding Speed: Normalized decode speeds show significant variance (CV > 50%). Longer contexts correlate with slower observed generation, particularly for Codex.

3. Tool Calling Behavior

• Heavy-Tailed Distribution: Tool usage is highly skewed. The top 3 tools (e.g., Bash, Read, Edit for Claude; exec_command, write_stdin, apply_patch for Codex) account for over 80% of all calls.
• Latency Variance: Tool latency ranges from milliseconds to hours. While calls longer than 1 minute constitute only ~4-5% of total calls, they account for 85-92% of total tool-call time.
• Overhead: For Codex, observed end-to-end tool latency often exceeds internal execution time, with a significant residual overhead (average 1.11s) attributed to scheduling, approval, and runtime bookkeeping.

4. Prefix Cache Efficiency

• High but Imperfect Hit Rates: The global prefix cache hit rate is high (~95.7%), but misses are costly. Most misses occur during user-initiated steps where human idle time exceeds the cache eviction threshold.
• Human-Induced Misses: When human thinking gaps exceed 5 minutes, hit rates drop significantly; after 1 hour, almost all steps miss the cache.
• Redundant Prefill: Only ~19% of append tokens are "fresh" (truly new). The remaining ~81% are redundant re-prefills of context that could theoretically be cached. The current system amplifies prefill by 5.3× compared to an ideal, eviction-free cache.
• Cost Impact: Eliminating cache misses caused by human thinking time could reduce append prefill by 45.9% and total API costs by approximately 12.8%.

Key Contributions

1. TraceLab Dataset: The release of the first large-scale, cross-provider trace of real coding-agent usage, containing ~4,300 sessions and ~350k LLM steps.
2. Workload Characterization: A comprehensive analysis revealing that coding agents differ fundamentally from chat or completion workloads, characterized by long autonomous loops, long contexts with short outputs, and heavy-tailed tool latencies.
3. System Insights: Identification of specific inefficiencies in current serving systems, such as the high cost of prefix reads, the impact of human gaps on cache eviction, and the overhead of frequent tool-LLM switching.
4. Open Resources: Release of the dataset, trace collection pipeline, and analysis code to facilitate further research.

Significance and Future Directions

The paper argues that understanding these specific workload patterns is a prerequisite for optimizing LLM serving systems for coding agents. The findings point to several concrete research directions:

• Tool Calling: Developing denser or fused tool invocations and lower-overhead approval paths to reduce scheduling latency.
• Prefill Optimization: Implementing append-length-aware prefill routing and adaptive kernel selection to handle the bimodal input structure.
• Cache Management: Designing semantic-aware tool-latency prediction for eviction policies and exploring cheaper KV-cache storage or prefetching strategies to handle human-paced gaps.
• Attention Mechanisms: Investigating sparse attention to reduce decoding costs associated with long prefixes.

The authors conclude that while existing systems provide mechanisms for serving agents, the unique characteristics of coding-agent workloads (persistent context, tool loops, human delays) require specialized optimization strategies that this trace aims to inform.