🤖 AI

Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

This paper establishes two critical gates for optimizing GPU execution in LLM-agent control—deadline-feasible cohort supply and observation placement—by demonstrating that specialized dynamic programming can significantly increase concurrent GPU work and that keeping route decisions on-device avoids costly host round trips while maintaining correctness.

Original authors: Josef Liyanjun Chen

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Josef Liyanjun Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, high-speed train station where thousands of tiny, automated robots are constantly arriving, asking questions, and waiting for instructions. In the world of artificial intelligence, these robots are "agents" that use giant brain-like computers (called GPUs) to think and then run small tools to get things done. But here's the catch: every time a robot finishes a thought, it has to run back to a central control room (the CPU) to ask, "What should I do next?" This back-and-forth trip is like a runner sprinting from the track to the coach's tent, getting a new instruction, and sprinting back. If the robots are small and fast, but the trip to the tent is slow, the whole system gets stuck in traffic. Scientists have long wondered: Can we get the coach to stay on the track with the runners? Can we group the robots together so they can all get their next instructions at once, without ever leaving the track? This is the puzzle of "agent control," and it's the difference between a smooth, super-fast race and a chaotic, slow-motion jam.

This paper, titled "Ready Cohorts," tackles that exact traffic jam by asking two simple but tricky questions: First, do enough robots ever show up at the same time to make a group trip worth it? Second, if they do, does keeping the decision-making inside the track (on the GPU) actually save time, or is it just a fancy trick that doesn't work?

The researchers set up two different experiments to find out. In the first part, they looked at a massive history of robot movements (a "trace" of 851 sessions) to see how many robots could be grouped together. They compared a standard method, which waits for a fixed time window (like a bus that leaves exactly at 5:00 PM regardless of how many people are there), with a smarter, "exact" method that waits just long enough to grab the perfect group. They found that the smart method could actually grab 43.00% of the robots, whereas the fixed-window method only managed 30.19%. That's a big difference! It means that by being flexible with timing, you can recover about 81.83% of the lost opportunities. However, they also discovered a hard limit: if the group size requirement is too high (specifically, if you need 256 robots to form a group), the system often fails to find enough robots in short time windows, especially when the total number of active robots is lower. In those cases, the "grouping" idea collapses, and the robots just have to go it alone.

In the second part of the study, the team tested the "stay on the track" idea. They built a simulation where a robot makes a binary decision (like "turn left" or "turn right"). They compared two ways of handling this: one where the decision is sent back to the host computer (the coach's tent) and then sent back to the robot, and another where the decision stays right there on the track (the GPU). The results were clear: keeping the decision on the track was always faster. Across four different types of computer hardware, the "stay on track" method was between 1.19 times and 2.39 times faster than the method that sent the decision back and forth. For example, on one specific setup, the fast method took about 258 microseconds, while the slow method took 467 microseconds.

However, the paper is very careful not to overhype this. They explicitly ruled out a few ideas. They tested a "nested" approach where the track tries to launch the next step without sending the decision back, but without actually removing the decision-making step. This failed; it was slower in every single test. This proves that the speedup doesn't come from just launching things faster on the track; it specifically comes from not having to send that tiny decision back to the coach's tent.

So, what's the bottom line? The paper suggests that we can speed up these AI agents, but only if two conditions are met. First, we need enough robots arriving at the same time to form a group (the "cohort supply"). Second, we must keep the decision-making right where the work happens, avoiding the slow trip back to the central computer. If either of these conditions isn't met, the fancy GPU tricks won't help, and the system is better off sticking to the old, reliable way. The author concludes that while the potential is there, building a real-world system that combines these two ideas is the next big challenge, and it needs to be tested with real-world traffic, not just simulations.

Technical Summary: Ready Cohorts – Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

Problem Statement

LLM-agent services perform deterministic control transitions between model and tool calls (e.g., routing outcomes, updating state, emitting effects). While individual transitions are small, high-concurrency services execute them continuously, creating a datacenter concern where host orchestration sits on the critical path, causing bursty CPU demand and repeated CPU–GPU boundary crossings.

The paper addresses two specific systems questions:

Cohort Supply: Under what conditions does the deterministic control path expose enough concurrent work to be profitably grouped and executed on a GPU?
Observation Placement: What is the performance cost of returning a GPU-computed decision to the host for redispatch versus keeping that decision on the device?

The author defines the "ready-cohort boundary" as the interface between sufficient concurrent work supply and the overhead of observation placement.

Methodology

The paper employs two distinct, numerically separated experiments to answer these questions, avoiding the conflation of workload characteristics with mechanism performance.

1. Trace Study: Cohort Supply Analysis

Data Source: A frozen, prospectively defined replay of the public Exgentic agent-trace dataset (tau2_airline, tau2_retail, tau2_telecom), comprising 851 sessions and 9,031 LLM spans.
Model: A stationary Poisson process generates session arrivals. Events are grouped by an "outcome-derived route key" (a proxy for executable compatibility, not a proof of semantic fusion).
Metrics:
- $F$ (Fixed-Partition Share): The fraction of events eligible for batching under fixed time windows.
- $P^\star$ (Exact Offline Share): The maximum fraction of events schedulable by an optimal offline algorithm with future knowledge, zero service time, and unlimited capacity. Computed via a specialized dynamic program for equal relative deadlines.
- $U$ (Local Upper Bound): The fraction of events that locally overlap with $K$ same-route events within their launch window.
- $K$ (Hardware Threshold): A safe suffix threshold where cohort sizes $\ge K$ are deemed profitable.
Configuration: The primary cell tests 100,000 target active sessions, $K=256$ , and a 50 ms launch deadline.

2. Mechanism Study: Observation Placement

Setup: A CUDA-based experiment comparing three mechanisms for executing a state transition and a binary route decision on a GPU.
- Host Round Trip: Launch predicate $\to$ copy result to host $\to$ synchronize $\to$ launch selected route graph.
- Device Resident: Launch a root graph where a device-side selector tail-launches the selected path graph, keeping the decision on-device for $H$ epochs.
- No-Decision Floor: A structural baseline omitting decision logic (oracle).
Grid: Four GPU placements (Local GTX 1660 Ti, Modal L4, RunPod L4, Lambda H100), varying agent counts ( $N$ ) and epoch horizons ( $H$ ).
Correctness: Verified against a separate host oracle for exact state equality and decision sequences across 14.5 million batched invocations.

Key Contributions

Ready-Cohort Formulation: The paper formalizes the boundary using distinct quantities: hardware threshold $K$ , fixed-partition share $F$ , exact offline share $P^\star$ , local bound $U$ , and online achieved share $A$ . It establishes the ordering $F \le P^\star \le U$ .
Exact Opportunity Instrument: A specialized dynamic program computes $P^\star$ exactly under zero service time and equal relative deadlines, providing a reproducible workload measurement without claiming algorithmic novelty.
Trace Evidence for Hidden Supply: The study demonstrates that exact sliding-deadline packing recovers significant opportunity missed by fixed windows.
Cross-Provider Mechanism Evidence: Empirical proof that retaining a binary decision on the device is consistently faster than a host-mediated round trip across diverse hardware and cloud providers.

Results

Cohort Supply (Trace Study)

In the primary cell ( $C=100,000$ , $K=256$ , $\delta=50$ ms):

Fixed-Partition Share ( $F$ ): 30.19%
Exact Offline Share ( $P^\star$ ): 43.00%
Local Upper Bound ( $U$ ): 45.85%
Gap Recovery: Exact packing recovers 81.83% of the opportunity lost at fixed window boundaries ( $P^\star - F$ ).
Boundary Sensitivity: Cohort supply collapses to zero for $C \le 10,000$ at $K=256$ . Lowering $K$ significantly increases $P^\star$ (e.g., at $C=100,000$ , $P^\star$ rises from 43.00% at $K=256$ to 66.8% at $K=32$ ).

Observation Placement (Mechanism Study)

Performance Gain: The device-resident path is faster than the host round trip in all 36 configurations across four placements.
Speedup Ratios: Within-placement row-medians range from 1.19 $\times$ to 2.39 $\times$ .
- Primary cell ( $N=256, H=32$ ): 1.71 $\times$ (GTX), 2.39 $\times$ (Modal L4), 2.06 $\times$ (RunPod L4), 1.84 $\times$ (Lambda H100).
- Absolute latency: Device path (258–309 $\mu$ s) vs. Host round trip (467–625 $\mu$ s).
Negative Control: A "nested device graph" that launches child graphs from the device but retains the host decision epoch was slower in all 60 configurations. This confirms that the speedup comes from removing the host decision epoch, not merely device-side launching.

Significance and Claims

The paper establishes two measurable "gates" for a GPU-based agent control plane:

Deadline-Feasible Cohort Supply: A workload must supply enough same-group work before launch deadlines. Fixed windows miss significant schedulable work; exact packing reveals this headroom.
Observation Placement: A device-resident decision avoids measurable host-mediated overhead (copy, sync, branch, redispatch).

Modest Scope and Limitations:

The paper explicitly states that the trace threshold $K=256$ is a swept candidate, not a measured crossover for the resident policy.
The mechanism horizon ( $H=32$ ) is not inferred from the trace.
The two studies are not multiplied; the offline optimum assumes zero service time and unlimited capacity, while the mechanism study uses synthetic state.
The "route key" is a conditioning proxy, not proof of semantic fusion.
The paper does not measure the online achieved share ( $A$ ), CPU displacement, or end-to-end service-level objectives (SLOs).

Conclusion:
The results provide a workload budget and a mechanism test for a future online route compactor. The author argues that a viable system must pass both gates: it must convert offline headroom into online work ( $A$ ) and displace CPU without harming model service. If it cannot, the boundary rejects the GPU design; if it can, these quantities define where GPU-resident agent control belongs in a datacenter.