Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI

MORI is an agent serving system that improves throughput and reduces time-to-first-token by treating idleness as a continuous spectrum to dynamically rank and partition agentic workloads between GPU and CPU memory tiers, thereby overcoming the limitations of traditional eviction policies in handling variable tool-call durations.

The Gist

Imagine you run a busy restaurant kitchen (the GPU) with a very small, high-speed counter space where chefs work. You also have a large, slower pantry (the CPU) nearby.

In the old days, restaurants only served simple, one-off orders. But now, customers are ordering complex, multi-course "agent" meals. These meals involve the chef cooking a bit, then stepping away to wait for an oven to finish baking, or waiting for a delivery driver to drop off ingredients, then coming back to cook the next step.

The problem is that the chef's workspace (the KV Cache) gets cluttered with ingredients and notes for every step of the meal. If the kitchen gets too crowded, you have to throw ingredients away to make room for new orders. If you throw them away, you have to start from scratch when the chef returns, which wastes time.

The Problem: The "One-Size-Fits-All" Mistake

Previous systems tried to manage this kitchen with a simple rule: "If you haven't cooked in a while, move your ingredients to the pantry."

But this rule is flawed because it doesn't understand the type of waiting:

1. The Busy Wait: The chef is waiting for a pot to boil for 30 seconds. They will be back at the stove immediately. If you move their ingredients to the pantry, you waste time running back and forth.
2. The Long Wait: The chef is waiting for a delivery truck that might take 10 minutes. They won't be back soon. Keeping their ingredients on the crowded counter just blocks space for other chefs.

Old systems treated all waiting the same. They would sometimes kick out the chef who was about to return (wasting time reloading ingredients) and keep the chef who was gone for hours (wasting counter space).

The Solution: MORI (The Smart Kitchen Manager)

The paper introduces MORI, a new system that acts like a smart kitchen manager. Instead of just looking at when a chef last worked, MORI looks at how idle they are relative to everyone else.

Think of "idleness" not as a switch (On/Off), but as a dimmer switch.

• Low Idleness (Bright): The chef is busy, cooking fast, and will be back at the stove in seconds. MORI keeps them on the main counter (GPU).
• High Idleness (Dim): The chef is stuck waiting for a long time. MORI moves them to the pantry (CPU) to free up the counter.

How MORI Works (The Three Zones)

MORI organizes the kitchen into three zones based on how "busy" the chefs are:

1. The Hot Counter (GPU HBM): This is for the busiest chefs. Their ingredients are right there, ready to go.
2. The Pantry (CPU DRAM): This is for chefs who are waiting for a long time. Their ingredients are stored here. It takes a little time to bring them back to the counter, but it's worth it because the counter is free for others.
3. The Waiting Room: If the pantry is full, some chefs have to wait outside. If they get called back, they have to start their recipe from scratch because their ingredients were thrown away.

The Magic Trick:
MORI constantly checks the "dimmer switch" for every chef.

• If the counter is full, it kicks out the chef who is most idle (the one waiting the longest) to the pantry.
• If a spot opens up on the counter, it brings back the chef who is least idle (the one about to start cooking).
• It also makes sure that if a chef was in the pantry, they go back to the same kitchen station they came from, so they don't have to run to a different part of the building to find their stuff.

The Results

The researchers tested this system using real-world coding agents (like AI programmers) on powerful computers. They found that MORI was much better than the old methods:

• Faster Service: The kitchen could handle 20% to 71% more orders per hour.
• Less Waiting: Customers got their first bite of food 18% to 43% faster.
• No Waste: It stopped the kitchen from wasting time reloading ingredients for chefs who were actually about to return.

In a Nutshell

MORI is a smart scheduler that realizes "waiting" isn't all the same. By treating idleness as a spectrum (a sliding scale) rather than a simple yes/no, it keeps the fast, busy work on the super-fast hardware and moves the long-waiting work to the slower, cheaper storage. This keeps the system running smoothly without wasting expensive resources or time.

Technical Summary

Technical Summary: MORI – Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems

1. Problem Statement

Modern Large Language Model (LLM) serving systems are increasingly tasked with agentic workloads, where a single user session consists of a stateful sequence of model invocations interleaved with tool calls (e.g., file I/O, code execution, human interaction). Unlike traditional chat requests, these agentic programs accumulate a growing KV cache across steps due to prefix dependencies.

The core challenge arises from the mismatch between memory capacity and workload dynamics:

• Memory Constraints: The aggregate KV cache of concurrent programs often exceeds GPU High Bandwidth Memory (HBM) capacity, necessitating offloading to CPU DRAM.
• Unpredictable Latency: Tool-call durations vary by orders of magnitude (from milliseconds to minutes). Transferring KV cache between GPU and CPU incurs significant overhead.
• Ineffective Policies: Standard eviction policies like LRU (Least Recently Used) or static program-aware policies fail in this setting. They may evict a program that is about to resume inference (during a short tool-call gap) while retaining a program blocked on a long-running tool call, leading to unnecessary recomputation and GPU memory waste.
• Capacity Mismatch: A binary classification of programs as "busy" or "idle" is insufficient because the ratio of busy-to-idle programs fluctuates dynamically. A fixed partition of GPU vs. CPU capacity cannot adapt to these shifts, leading to underutilization of one tier and oversubscription of the other.

2. Methodology: MORI

The authors propose MORI (Memory Offloader with Relative Idleness), a program-aware scheduler designed to manage KV cache placement across a two-tier memory hierarchy (GPU HBM and CPU DRAM).

2.1 Core Insight: Idleness as a Relative Spectrum

Instead of treating idleness as a binary state, MORI defines it as a continuous, relative spectrum. The system ranks all active programs based on how likely they are to remain GPU-resident.

• The Metric: MORI calculates an Idleness score ($\iota$) for each program using a sliding window of the most recent $k$ steps (inference + tool calls):
  $$\iota = \frac{T_{acting}^{(k)}}{T_{reasoning}^{(k)} + T_{acting}^{(k)}}$$
  Where $T_{acting}$ is the time spent in tool-call gaps and $T_{reasoning}$ is the time spent in GPU inference.
  • A score near 1 indicates an idle phase (dominated by long tool calls).
  • A score near 0 indicates a busy phase (rapid succession of short tool calls).
• Robustness: The windowed average allows the system to react quickly to phase transitions while smoothing out isolated outliers (e.g., a single slow command during a busy phase).

2.2 System Architecture

MORI operates as a scheduler between agent clients and a pool of LLM inference engine replicas. It manages a three-tier queue structure:

1. GPU Queue (HBM): Holds "busy" programs (low idleness). Their KV cache resides in GPU memory for immediate inference.
2. CPU Queue (DRAM): Holds "idle" programs (high idleness). Their KV cache is offloaded to CPU DRAM. These programs are paused until promoted back to the GPU.
3. Waiting Queue: Holds programs whose KV cache has been evicted entirely. They must recompute the full prefix upon resumption.

2.3 Scheduling Policy

MORI employs sticky placement to minimize the cost of offloading and reloading:

• Demotion: When GPU capacity is exceeded, the scheduler demotes programs with the highest idleness (most likely to remain blocked) to the CPU queue.
• Promotion: When GPU capacity frees up, the scheduler promotes programs with the lowest idleness (most likely to resume inference soon) from the CPU queue.
• Admission Control: Both GPU and CPU tiers enforce capacity limits. If both are full, new or waiting programs are placed in the Waiting queue.
• Affinity: In multi-replica deployments, MORI preserves cache affinity. If a program's KV cache is offloaded to a specific replica's CPU DRAM, the scheduler routes the program back to that same replica upon promotion, avoiding cross-replica recomputation.

2.4 Engine Integration

MORI integrates with the inference engine (implemented on top of ThunderAgent and SGLang) by propagating type labels (busy, idle, inactive) to the engine's KV cache tree nodes. The engine's eviction policy uses these labels as a high-priority sort key, ensuring that within the GPU tier, "busy" blocks are retained over "idle" ones, and vice versa for the CPU tier.

3. Key Contributions

The paper claims the following contributions:

1. Characterization of Agentic Workloads: Analysis of real-world coding agent traces reveals a persistent two-phase structure (busy vs. idle) that can be exploited by schedulers, despite the unpredictability of individual tool-call durations.
2. Continuous Idleness Metric: Introduction of a relative idleness metric that enables hardware-adaptive KV cache placement, moving beyond binary classifications.
3. MORI System Design: A scheduler that dynamically places programs across GPU HBM and CPU DRAM based on relative idleness, enforcing admission control at both tiers and maintaining replica affinity.
4. Performance Evaluation: Empirical validation showing significant throughput and latency improvements over existing baselines across diverse hardware and model configurations.

4. Experimental Results

The authors evaluated MORI on SWE-bench Pro traces using Claude Code, across four hardware/model configurations (H200 80GB, H200, B200; 7B, 30B MoE, 70B models) and both single- and multi-replica (DP=3) deployments.

4.1 Single-Replica Performance

• Throughput: At high concurrency (80 programs), MORI achieved 20–71% higher output throughput compared to the best offloading baseline (ThunderAgent + Offloading, which uses context-length-based eviction).
• Latency (TTFT): MORI reduced Time-To-First-Token by 18–43% compared to the offloading baseline and up to 2.8x compared to non-offloading systems.
• Scalability: While baselines suffered throughput collapse or plateaued as memory pressure increased, MORI maintained monotonic growth by proactively offloading idle programs and prioritizing busy ones for GPU residency.

4.2 Multi-Replica Performance

• Throughput: In a DP=3 setting, MORI achieved 54–79% higher throughput than the offloading baseline at 80 programs per replica.
• GPU Utilization: MORI sustained 99%+ GPU utilization, whereas phase-oblivious schedulers dropped to 59–76% due to inefficient memory usage (keeping idle programs on GPU).
• Affinity & Churn: MORI drastically reduced program churn (backend switching). Only 0.3–2.9% of programs switched backends compared to 14–15% in baselines, preserving KV cache locality and avoiding recomputation.

5. Significance and Claims

The paper positions MORI as a necessary evolution in LLM serving systems to handle the emerging paradigm of agentic workloads.

• Shift in Scheduling Unit: It argues that the unit of scheduling must shift from individual requests to agentic programs to capture the persistent memory working set and prefix dependencies.
• Adaptive Resource Management: By treating idleness as a relative spectrum, MORI solves the problem of mismatched capacity ratios without requiring per-hardware tuning. It effectively utilizes finite GPU and CPU memory by aligning placement with the behavioral state of the program rather than static properties.
• Practical Impact: The system demonstrates that coordinated offloading, driven by runtime behavioral signals, can significantly improve throughput and latency in complex, stateful agent scenarios where traditional LRU or static policies fail.

The authors note that while the current implementation focuses on a two-tier hierarchy, the design naturally generalizes to additional tiers (e.g., SSD or remote DRAM) and heterogeneous hardware clusters.