🤖 AI

A Policy-Driven Runtime Layer for Agentic LLM Serving

This paper proposes a new architectural "agent runtime layer" that bridges the gap between multi-agent frameworks and LLM serving engines to enable policy-driven optimizations, demonstrating through the CacheSage system that this approach significantly improves cache hit rates, time-to-first-token, and throughput across diverse multi-agent workloads.

Original authors: Rui Zhang, Chaeeun Kim, Liting Hu

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rui Zhang, Chaeeun Kim, Liting Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy, high-end restaurant.

The Current Setup: A Communication Breakdown
Right now, your restaurant has two distinct layers that don't talk to each other well:

The Head Chef (The Agent Framework): This person knows the menu, the roles of the waiters, and the specific instructions for each table. They know who is ordering and what they need. However, they never see the kitchen floor; they don't know which pots are currently on the stove or which ingredients are running low.
The Kitchen Staff (The Serving Engine): This team sees every single order that comes in. They know exactly how many pots are boiling and how fast they can cook. But they have no idea who the customers are or what the "story" of the meal is. To them, every order is just a generic request.

The Problem: The "Seam" Where Things Break
Because these two groups don't share information, the restaurant makes inefficient decisions.

Example: The Head Chef knows that Table 4 always orders the same appetizer before their main course. But the Kitchen Staff doesn't know this. So, every time Table 4 orders, the kitchen has to start chopping onions from scratch, even though they just did it five minutes ago for the same table.
The paper calls this the "seam." Currently, if you want to fix this, you have to patch the Head Chef's notes or the Kitchen's workflow with a specific, one-off rule. It's messy and doesn't scale.

The Solution: The "Agent Runtime Layer" (The New Floor Manager)
The authors propose building a third layer right between the Chef and the Kitchen: a Floor Manager.

This Floor Manager has a special job. They listen to the Chef (to know the roles and identities) and watch the Kitchen (to see the cooking events). They use this combined knowledge to make smart decisions using four simple tools:

Observe: "I see a new order coming in from the 'Planner' waiter."
Score: "Based on history, this 'Planner' order is very important and likely to be followed by a 'Coder' order. Give it high priority."
Predict: "I bet the next order will be from the 'Coder' waiter. Let's get their ingredients ready now."
Act: "Go ahead and pre-warm the stove for the 'Coder' so there's no delay."

This Floor Manager acts as a universal translator. Any new rule (like "be fair to all tables" or "save energy") can plug into this manager without breaking the Chef or the Kitchen.

The Case Study: "CacheSage" (The Smart Pantry)
To prove this works, the authors built a specific Floor Manager called CacheSage to handle the "pantry" (the computer's memory, or KV cache).

The Old Way: The kitchen throws away ingredients (memory) based on how long ago they were used. If the "Planner" waiter comes back after a break, the kitchen has to re-chop everything because the ingredients were tossed out.
The CacheSage Way: The Floor Manager learns the patterns. They notice that "Planner" almost always leads to "Coder."
- When the "Planner" finishes, the Floor Manager says, "I predict 'Coder' is next."
- They keep the "Planner's" ingredients safe (so they don't get thrown out) and even start prepping the "Coder's" ingredients before the order even arrives.

The Results
When they tested this on five different real-world "restaurant" scenarios (complex AI tasks):

Less Waste: They kept the right ingredients in the pantry 13% to 37% more often than before.
Faster Service: Customers got their food 12% to 29% faster because the kitchen didn't have to start from scratch.
More Customers: The restaurant could serve 6% to 14% more tables per hour.

In Summary
The paper argues that to make AI agents run efficiently, we can't just tweak the top layer (the logic) or the bottom layer (the hardware). We need a dedicated "middle manager" that understands both the identity of the agents and the events of the engine, using a simple set of four rules to make the whole system smarter and faster.

Technical Summary: A Policy-Driven Runtime Layer for Agentic LLM Serving

1. Problem Statement

Multi-agent Large Language Model (LLM) systems have become the dominant production workload, yet the existing serving stack is architecturally misaligned with their requirements. Current systems operate on a two-tier structure:

Agent Framework (Top): Systems like AutoGen, LangGraph, or MetaGPT that manage agent identities, role schemas, and dispatch logic. They emit LLM API calls but remain blind to engine-level events (e.g., token generation, KV cache state).
Serving Engine (Bottom): Engines like vLLM or SGLang that handle prefilling, batching, and token generation. They observe every event but possess no knowledge of agent identities or the semantic structure of the multi-agent workflow.

This "seam" creates a critical gap for cross-cutting policies. Many optimization strategies—such as KV cache eviction, batch shaping, speculative execution, fairness, and safety enforcement—require simultaneous access to agent identity metadata (from the framework) and engine-level events (from the serving engine). Currently, these policies are implemented as "one-off patches" into either the framework or the engine. This approach is brittle, non-modular, and fails to leverage the full context required for optimal decision-making.

2. Methodology: The Agent Runtime Layer

The authors propose an architectural intervention: inserting a third tier, the Agent Runtime Layer, between the framework and the serving engine. This layer acts as a substrate that owns the cross-cutting agent metadata and exposes a stable, policy-agnostic interface.

The Four Primitives

The layer defines four generic primitives through which any agent-aware policy can plug in. These primitives form an observe → score → predict → act feedback loop, unified by agent identity as the shared coordinate:

observe(event): The runtime receives policy-relevant events from neighbors (e.g., block touches, request arrivals, tool returns). It updates internal state tagged by agent identity.
score(item): The runtime returns a per-item priority used by a neighbor to rank decisions (e.g., which KV block to evict, which request to admit).
predict(horizon): The runtime returns a probabilistic forecast of future agent activity (e.g., the next agent role, next tool call).
act(side_effect): The runtime issues off-critical-path side effects to a neighbor (e.g., warming a cache, pre-loading weights, hedging a speculative call). This is the only primitive that mutates neighbor state.

Case Study: CacheSage

To validate this abstraction, the authors instantiate the layer for KV caching, the policy with the largest immediate impact on serving costs. They introduce CacheSage, which addresses the problem of reactive prefix caches dropping "agent anchors" (system prompts and tool definitions) before the next session arrives.

CacheSage implements the four primitives via three components:

Transition Learner (observe): Maintains a sliding window of pairwise agent transition counts ( $n(a', a)$ ) to compute a first-order Markov transition matrix online. It identifies the "agent anchor" (system + tools) via content hashing.
Survival-Probability Scorer (score): Approximates the probability that an agent will fire within the next $K$ steps. Instead of expensive exact calculations, it uses a thresholded Directed Acyclic Graph (DAG) and Breadth-First Search (BFS) to compute a "hop count" proxy. This score is combined with the engine's existing recency clock to determine eviction priority.
Cross-Session Prefetch (predict + act): Between scheduler steps, the runtime predicts the next agent ( $a^*$ ) and issues a "warmup" request. This request contains only the system prompt and a minimal user message to populate the KV cache for the predicted agent's anchor without injecting full session-specific content.

3. Key Contributions

The Agent Runtime Layer: A proposed third tier for the serving stack that exposes a four-primitive interface (observe, score, predict, act). The authors map nine distinct agent-aware policies (including KV caching, tool-result memoization, batch shaping, and fairness) onto this interface, arguing that they all share the same architectural need for agent-identity-tagged events.
CacheSage: A concrete implementation of the layer for KV caching. It learns per-workload agent transition matrices online and uses them for survival-based eviction and between-step prefetching. It operates with bounded state ( $\le$ 20 KB per workload) and includes a graceful fallback to standard LRU if no structure is learned.
Empirical Validation: Preliminary results on five real-world multi-agent workloads (MMLU, MT-Bench, GAIA, GSM8K, HumanEval) demonstrating the efficacy of the approach.

4. Results

Experiments were conducted on NVIDIA H100 GPUs using Llama-3.1-8B-Instruct, comparing CacheSage against vLLM and Continuum.

Cache Hit Rate: CacheSage achieved a +13 to +37 percentage point (pp) lift in cache hit rates compared to vLLM across all five workloads. This is attributed to retaining agent anchors across session boundaries that reactive caches typically evict.
Latency: Mean Time to First Token (TTFT) and end-to-end per-turn latency were reduced by 12% to 29%.
Throughput: System throughput increased by 6% to 14%.
Overhead: The hot-path overhead is negligible. The observe function runs in $\sim1 \mu s$ per block touch, and the scorer runs in single-digit microseconds. The coordinator state remains under 25 KB.

Notably, the baseline Continuum system, which uses TTL pins, actually reduced hit rates by 9–23 pp on three real-task workloads, highlighting the trade-offs of static policies versus the adaptive nature of the runtime layer.

5. Significance and Claims

The paper claims that the "seam" between the agent framework and the serving engine is best addressed by an architectural change rather than incremental point fixes.

Modularity: The four-primitive interface allows distinct policies to share the same observation stream and compose against the same primitives without requiring new engine surgery for each policy.
Generality: The layer is framework-agnostic (supporting AutoGen, LangGraph, etc.) and engine-portable, requiring only specific hooks rather than deep internal modifications.
Validation of the Gap: By demonstrating significant performance gains on KV caching, the authors argue that the architectural gap is real and that an agent runtime layer is a necessary component for efficient agentic LLM serving.

The authors remain modest regarding the scope of their current work, noting that while CacheSage validates the abstraction, the other eight policies mapped to the layer (e.g., tool-result memoization, speculative agent execution, fairness) remain open challenges for future work. The primary contribution is the interface and the architectural argument, not the completion of every policy implementation.