Technical Summary: Observation, Not Prediction – Conversation-Level Disaggregated Scheduling for Agentic Serving

1. Problem Statement

Existing Large Language Model (LLM) serving systems are designed around the "request" as the atomic scheduling unit, a model suited for single-shot inference where a prompt is prefilled and decoded in one go. However, agentic workloads break this contract. An agent's task is resolved through a sequence of dependent inference calls interleaved with tool use, environmental feedback, and state accumulation.

This shift creates a scheduling–value mismatch:

• System View: Optimizes per-turn latency (Time-to-First-Token, TTFT) for every intermediate turn, many of which are internal tool calls invisible to the user.
• User View: Waits for the conversation's final, externally meaningful output. The relevant metric is Time-to-First-Effective-Token (TTFET)—the latency until the first token of genuine user-visible progress.

Current multi-turn systems attempt to optimize this by deciding, turn-by-turn, whether to disaggregate the prefill (compute-bound) from the decode (memory-bound) phase. To make this decision, schedulers must predict unobservable quantities: how many tokens a turn will decode, whether it will call a tool, and how much the Key-Value (KV) cache will grow. This reliance on prediction introduces structural brittleness; any prediction error leads to suboptimal routing, queueing delays, and degraded energy efficiency.

2. Methodology: ConServe

The paper introduces ConServe, a system that raises the scheduling unit from the turn to the conversation. The core insight is that while agentic conversations exhibit irregularity at the turn level, they possess a stable, two-phase structure at the conversation level:

1. Phase 1 (Compute-Bound): A single, heavy prefill of the initial long task description (turn 1).
2. Phase 2 (Memory-Bound): A long tail consisting of the first decode and all subsequent turns, which involve short append-prefills and decoding over an accumulating KV cache.

Key Design Principles

• Observation over Prediction: ConServe eliminates the need to predict turn-level outcomes (e.g., future decode lengths or tool behaviors). Placement decisions are based entirely on directly observable state:
  • The input token count of the first-turn prompt (determines prefill load).
  • The current KV cache occupancy of decoder replicas (determines memory-bound load).
• Conversation-Level Pinning:
  • The first-turn prefill is routed to a dedicated, high-throughput prefiller node.
  • The resulting KV cache is transferred exactly once to a selected decoder replica.
  • The conversation is pinned to that specific decoder for its entire lifetime. All subsequent turns (append-prefills and decodes) execute locally on that decoder, reusing the KV cache without further network transfers.
• Reactive Scheduling: The scheduler admits new conversations based on current system capacity (prefill input rate vs. prefiller throughput; total KV occupancy vs. decoder pool capacity). It does not migrate conversations once placed.
• Heterogeneous Mapping: The two-phase structure naturally maps to heterogeneous GPU tiers. The compute-bound first-turn prefill runs on high-power GPUs, while the memory-bound tail runs on lower-power GPUs with sufficient memory bandwidth, as the tail's performance is gated by memory rather than raw compute.

3. Key Contributions

1. Conversation-Level Scheduling Paradigm: The paper identifies that the scheduling unit (turn) is the source of brittleness in agentic serving. By shifting to the conversation, the system replaces prediction with observable state, eliminating the need for learned cost models or complex predictors.
2. ConServe System: A minimal instantiation of this principle that routes first-turn prefills to high-power GPUs, transfers KV state once, and pins the conversation to a single decoder. It optimizes for TTFET and energy efficiency without learned models.
3. Empirical Demonstration of Structural Brittleness: The authors show that per-turn scheduling (e.g., in baselines like AMPD) suffers linearly from prediction errors. As the error rate increases, SLO violations and energy inefficiency degrade linearly. In contrast, ConServe's performance remains constant by construction because it makes no per-turn routing decisions.
4. Energy Efficiency via Heterogeneity: The paper demonstrates that the conversation-level phase split allows for tiered GPU allocation without additional machinery, significantly improving energy efficiency.

4. Experimental Results

Evaluated on agentic traces (SWE-bench) using a Qwen3-0.6B model on an NVIDIA A40 cluster, ConServe was compared against collocated, fully disaggregated, and per-turn prediction-based (AMPD) baselines.

• Latency (TTFET): ConServe reduces the p95 TTFET by 51.08% compared to the per-turn prediction baseline (AMPD). It achieves this by avoiding the queueing delays caused by misrouted turns in the prefiller.
• SLO Adherence: ConServe sustains zero SLO violations across all tested request rates, including saturation points. Baselines fail at saturation: collocated systems suffer from prefill-decode contention, and per-turn systems accumulate routing errors.
• Energy Efficiency:
  • In homogeneous settings, ConServe improves energy efficiency by 7.51% over the per-turn baseline.
  • In heterogeneous settings (300W prefiller, 200W decoders), ConServe achieves an additional 22.75% gain in energy efficiency (tokens per Joule) while maintaining latency performance. This is because the memory-bound tail is insensitive to the reduced compute power of the capped GPUs.
• Robustness: Unlike AMPD, whose performance degrades linearly with prediction error rates (e.g., 10% error leads to significant SLO violations), ConServe is immune to such errors as it relies on observable state.

5. Significance and Claims

The paper claims that the fundamental abstraction for agentic serving is the conversation, not the turn. By recognizing the stable two-phase structure of agentic workloads, ConServe demonstrates that:

• Placement should be reactive, not predictive. Relying on observable state (input length, KV occupancy) is sufficient and more robust than predicting future turn behaviors.
• Scheduling granularity dictates system stability. The fragility of current multi-turn systems is not an inherent property of the workload but a consequence of scheduling at the turn level.
• Energy efficiency is a natural corollary. The conversation-level phase split enables efficient heterogeneous mapping without complex orchestration, spending high-power compute only where necessary (the initial prefill) and utilizing lower-power hardware for the memory-bound tail.

The authors present ConServe not as a complex, learned system, but as a "minimal instantiation" of a principled paradigm that aligns system optimization with the actual user value (conversation completion) rather than intermediate system metrics (per-turn latency).