💻 computer science

Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics

This paper characterizes cross-instance attention in frontier LLMs by demonstrating that routing compressed query vectors is often more efficient than moving KV-cache blocks across GPU fabrics, and provides a validated topology-aware cost model and decision predicate to optimize this choice on real multi-node H100 clusters.

Original authors: Bole Ma, Jan Eitzinger, Harald Köstler, Gerhard Wellein

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bole Ma, Jan Eitzinger, Harald Köstler, Gerhard Wellein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The Library vs. The Reader

Imagine a massive library (the AI's knowledge base) that is so huge it doesn't fit in one building. It's spread across several different library branches (different GPUs).

Now, imagine a reader (the AI processing a question) is sitting in Branch A. They need to look up a specific fact that is stored in a book sitting on a shelf in Branch B.

The Old Way (Moving the Cache):
In the past, the standard solution was to send a truck to Branch B, load the entire book onto the truck, drive it back to Branch A, and then let the reader read it.

The Problem: Books are heavy and bulky. Even if the reader only needs one sentence, you have to move the whole book. If the library is huge, this trucking process takes forever and clogs the roads.

The New Idea (Moving the Query):
This paper suggests a smarter approach: Instead of sending a truck to get the book, send a tiny, lightweight note (the "Query") to Branch B. The note says, "Please look up this specific sentence in your book and write the answer on a postcard."

The Benefit: The note is tiny (about 1 kilobyte). The book is huge (millions of bytes). It is much faster to mail a postcard to the library and get a reply than to drive a truck back and forth.

The Secret Sauce: "MLA" (The Compressed Book)

Why does this work now when it didn't before? The paper focuses on a specific type of AI architecture called Multi-head Latent Attention (MLA).

Think of MLA as a special compression technique. In older AI models, the "book" (the data) was huge and unwieldy. But with MLA, the AI compresses every page of the book into a tiny, dense summary.

Because the data is so compressed, the "book" is still big, but the "sentence" the reader needs to look up is incredibly small.
This creates a massive imbalance: The Query (the note) is tiny, but the Cache (the book) is still large. This makes sending the note the obvious winner.

The Experiments: Testing the Roads

The researchers didn't just guess; they tested this on real, high-speed supercomputers (using NVIDIA H100 chips). They looked at two main things:

The Road Type (The Network): They tested different "roads" connecting the computers, from fast local cables (NVLink) to cross-building fiber optics (InfiniBand).
- Finding: Even on the fastest roads, moving the big book is slow because of the "loading time" (getting the book ready to move). Moving the tiny note is fast because it's just a quick trip.
- Surprise: On very fast roads, the speed of the road didn't matter as much as the speed of the truck driver (the computer's ability to start the transfer). A single note travels just as fast on a slow road as on a super-fast road because the note is so small it doesn't need the whole road.
The "Splice" Tax:
- When you move a book from one branch to another, you often have to re-bind it or adjust the pages so they fit the new shelf. The paper calls this a "splice." It takes about 3 milliseconds (a long time in computer terms) just to prep the book.
- Moving the note avoids this tax entirely. The note arrives, gets answered, and is gone.

The Rules for the AI Manager

The paper gives a simple set of rules for the "manager" of the AI system to decide what to do:

Rule 1: At the start of a conversation (Decoding), always send the note.
If the AI is answering a question step-by-step, the "note" is so small and the "book" so big that sending the note is 60 to 100 times faster than moving the book.
Rule 2: Only move the book if you're going to read it a lot.
If the AI is going to need that same book for a long time in the same location, it might be worth moving the book once and keeping it there. But for quick, one-off questions, send the note.
Rule 3: Don't worry about the road speed.
For these tiny notes, a slow road is almost as good as a fast road. The bottleneck isn't the road; it's the time it takes to write the note.

The "Agent" Scenario

The paper mentions a specific use case: Agentic Workloads. Imagine a team of digital assistants (agents) all trying to read the same giant code manual or legal contract.

Old Way: Every time an agent asks a question, the system tries to pull the relevant chunk of the manual to that agent's computer.
New Way: The system sends the agent's question to the computer holding the manual. The computer answers it and sends the tiny result back. The manual stays put. This allows hundreds of agents to ask questions simultaneously without clogging the network.

Summary

The paper proves that for modern, compressed AI models, it is almost always faster to send the question to the data than to send the data to the question.

They built a mathematical formula (a "cost model") that tells computer systems exactly when to do this. It turns out that for the tiny, fast-paced questions AI asks while thinking, "moving the query" is the clear winner, saving massive amounts of time and energy.

Technical Summary: Move the Query, Not the Cache

Problem Statement
As Large Language Models (LLMs) scale, serving workloads increasingly rely on Multi-head Latent Attention (MLA) and sparse attention mechanisms. In these architectures, the Key-Value (KV) cache is compressed into narrow latent vectors or selected via sparse indexers, creating a "byte asymmetry": a single query row (e.g., $\approx 1$ KB) is orders of magnitude smaller than the chunk of KV cache it attends to (e.g., megabytes).

When the corpus of interest (a canonical codebase, legal documents, or a shared enterprise knowledge base) exceeds the High Bandwidth Memory (HBM) of a single GPU, it must be partitioned across multiple instances. This necessitates cross-instance attention, where a decoding query on one GPU must access KV blocks residing on another. The prevailing industry reflex is to move the cache (fetch): pulling the remote KV blocks to the requester, splicing them into the local cache, and attending locally. However, this approach incurs a significant "position-adaptation" cost (re-rotating cached positional encodings to the new offset) and a large data transfer. The paper posits that under MLA and sparse selection, the arithmetic may invert: it might be cheaper to route the query to the holder, compute partial attention there, and merge the small result back.

Methodology
The authors characterize this trade-off on a real multi-node H100 cluster connected via NDR-200 InfiniBand with device-initiated RDMA (IBGDA) enabled. Their methodology involves:

Cost Decomposition: They model cross-instance redistribution as an additive cost function: $T_{redist} = T_{probe} + T_{transfer} + T_{compute} + T_{return} + T_{merge}$ .
Primitive Comparison: They compare three primitives:
- Fetch: Pull the entire KV chunk, adapt positions (splice), and attend locally.
- Route: Send the query to the holder, compute partial attention, and merge the result via online softmax.
- Local: Recompute the chunk from scratch (re-prefill).
Hardware Characterization: They measure the latency and bandwidth of IBGDA for attention-shaped payloads (kilobyte-scale query batches) and compare them against CPU-proxy paths. They also analyze the impact of topology (same-leaf vs. spine-traversing), fabric type (NVLink, PCIe, InfiniBand), and concurrency.
Correctness Verification: They verify that the routed partial attention, when merged via online softmax, reproduces single-instance attention results within bf16 noise floors, even for sparse selection scenarios.

Key Contributions

Topology-Aware Cost Model: The authors derive a closed-form cost model for cross-instance attention that fits real H100 IBGDA measurements with $\approx 7\%$ Mean Absolute Percentage Error (MAPE) for batched dispatch ( $M_q \ge 512$ ). The model identifies that the cost is dominated by a fixed probe latency ( $\approx 16 \mu s$ ) and a payload-independent effective bandwidth ( $\approx 25$ GB/s for single-block dispatch), rather than peak link bandwidth.
Device-Initiated RDMA (IBGDA) Regime Characterization: Unlike prior work on MoE dispatch (where CPU-proxy paths sometimes win for tiny messages), this paper establishes that for attention payloads ( $\approx 2$ KB), device-initiated RDMA is superior, reducing per-fetch latency by 40% compared to host-mediated paths.
Closed-Form Selection Predicate: They provide a decision rule for schedulers to choose between route, fetch, or local based on request shape ( $M_q$ $M_{q}$ ), chunk size ( $c_t$ $c_{t}$ ), and fabric characteristics.
- Route wins at decode (small $M_q$ ) because it avoids the $\approx 3$ ms position-adaptation splice required by fetch and avoids the compute cost of local.
- Fetch only becomes viable when amortizing the splice cost over many subsequent local steps.
Scatter and Selection Analysis: For sparse attention (e.g., DeepSeek-V3.2, GLM-5.1), the paper demonstrates that fetch becomes a scattered gather across multiple holders, incurring a penalty that grows with the number of holders. Route remains flat in cost because the query is sent once, and the holder attends to the scattered blocks in place without re-rotation.
Holder Capacity Elbows: They identify specific bottlenecks for the holder side:
- Copy Elbow: A pool of $K=8$ CUDA streams is optimal for staging data for concurrent requests.
- Compute Elbow: A holder can serve $\approx 8$ concurrent routed requesters with negligible latency increase before compute saturation occurs.

Results

Latency Dominance: At typical decode batch sizes ( $M_q = 256$ ), a routed round trip costs 31–48 $\mu s$ across various fabrics (NVLink, PCIe, IBGDA). This is 60–100 $\times$ faster than the $\approx 3$ ms splice cost of fetching a chunk.
Wire Byte Efficiency: Routing moves $\ge 76\%$ fewer wire bytes than fetching at decode batches.
Fabric Independence: At decode sizes, the routed dispatch is dispatch-bound (limited by the ability of a single thread block to issue transfers, $\approx 18\text{--}25$ GB/s) rather than bandwidth-bound. Consequently, a 900 GB/s NVLink 4.0 link performs similarly to a cross-node 25 GB/s InfiniBand link for single-block queries; peak bandwidth only matters for large batch sizes.
Congestion Resilience: The routing primitive is robust to network congestion. Latency remains flat until the link is fully saturated ( $K=3$ concurrent flows), and even then, it remains an order of magnitude faster than fetching.
Host Overhead Caveat: While routing wins on wire bytes and transport latency, the authors note that in their current prototype, host overhead (Python client, dispatch logic) dominates the end-to-end time-to-first-token. At small batch sizes, a simple "bytes-back" fetch can currently win end-to-end due to lower host overhead, though three specific implementation reductions (collapsed response, compute amortization, batching) are identified to close this gap.

Significance and Claims
The paper claims that the "move-the-cache" reflex is suboptimal for the emerging regime of compressed and sparse attention. By inverting the arithmetic, routing the query becomes the dominant primitive for cross-instance attention, particularly for agentic workloads where many sub-agents query a shared, immutable corpus.

The authors emphasize that their contribution is not a new routing algorithm (which exists in prior work like DistAttention) but a characterization and cost model that validates the economic case for routing under specific hardware and architectural conditions (MLA, IBGDA, sparse selection). They argue that this byte-asymmetry is the key property that motivates device-initiated RDMA for attention. The paper concludes that while routing is theoretically superior, realizing its full end-to-end latency benefits in a production serving stack requires addressing host-side overheads, a task left for future engineering efforts. The provided cost model and predicate serve as reusable artifacts for system designers to make topology-aware scheduling decisions without needing to re-profile every new configuration.