🤖 AI

Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory

Nexus accelerates agentic LLMs on unified memory by decoupling tool routing from expensive schema prefilling via a semantic lookaside buffer and compressed signatures, while employing a depth-adaptive KV-cache splicing mechanism with fallback re-decoding to maintain output fidelity despite rotary position embedding drift.

Original authors: Mustafa Arslan

Published 2026-08-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mustafa Arslan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a specific type of software agent is learning to act on our behalf. These digital assistants do not just answer questions; they reach out to use tools, like checking a calendar, searching a database, or sending an email. To do this, the software must first understand a detailed instruction manual for every single tool it might use. As the number of available tools grows into the hundreds, these manuals become enormous, filling up the computer's short-term memory with repetitive text before the agent can even begin to think. This creates a bottleneck: the more tools the agent has, the longer it takes to start working, because the computer must re-read and re-process these massive instruction sets every single time it takes a new step.

Researchers have been looking for a way to bypass this bottleneck. One idea was to save the computer's work on these instructions in a compressed form, like a bookmark, and simply paste it back into the memory whenever needed. This sounds efficient, but it runs into a fundamental problem with how modern AI models track time and order. These models use a system to remember where they are in a sequence of words, and if you move a saved chunk of work to a different spot in the memory, the model gets confused about the order of events. It is like taking a page from the middle of a book and pasting it into a different chapter; the story might still make sense, but the subtle connections between sentences can break, leading to errors.

A team of independent researchers has now built a system called Nexus to navigate this tricky landscape. They discovered that while you cannot simply move these saved instruction blocks around without risk, you can do so if you are careful about where you place them and how you fix the errors that inevitably creep in. Their work, tested on a specific type of powerful computer chip found in modern laptops, reveals a precise boundary for how far you can move these blocks before the AI's output becomes unreliable. They found that if the block is moved too far, the model's understanding of the sequence drifts, but this drift is predictable. The researchers designed a method to detect when this drift gets too large and automatically re-reads just the necessary part of the instructions to stitch the memory back together perfectly.

The most significant finding from this work is that the system does not need to rely on these risky memory shortcuts for its most critical task: deciding which tool to use. Instead of forcing the AI to read the massive instruction manuals to make a choice, Nexus uses a separate, ultra-fast lookup system. This system scans a compact list of tool names and descriptions to find the right match in microseconds. Once the tool is chosen, the AI generates the necessary arguments using a tiny, compressed summary of the instructions—often just a few dozen words—rather than the full, bloated text. This approach keeps the main memory clean and allows the agent to start its work much faster, finding the first word of its answer in less than half the time it would take to re-read the full manuals.

The researchers also rigorously tested the limits of their memory-splicing technique. They confirmed that while the method works well for short distances, there is a hard limit. If the saved block is placed more than 256 positions away from where it was originally created, the errors become too significant to ignore. At this point, the system stops trying to patch the memory and instead falls back to the slower, but safer, method of re-reading the full text. This ensures that the final output is always exactly as accurate as if the computer had never tried to shortcut the process. They also proved that a simpler, cheaper method of guessing when to stop the shortcut—based on a quick check of the memory's internal state—does not work, as it fails to predict the errors reliably.

In their tests, the Nexus system managed to handle a registry of 250 different tools without slowing down, maintaining a high success rate in choosing the correct tool. When the context was moderate, the system was up to 1.7 times faster than the traditional method of re-reading everything. However, as the conversation grew longer and deeper, the speed advantage naturally faded, eventually matching the performance of the standard method. This is not a failure, but a feature of their design: the system prioritizes getting the answer right over being fast. It guarantees that the output is never worse than the standard method, even if it sometimes takes the same amount of time.

The work was conducted on a single type of computer chip with a unified memory architecture, which allows the processor to access data directly without moving it between different parts of the machine. This specific hardware setup was essential for the memory-splicing technique to work, as it requires direct physical access to the memory cells. The researchers are clear that while the speed numbers are specific to this one setup, the underlying principles—the limits of moving memory blocks and the necessity of a separate routing system—appear to be universal truths about how these models function. They have provided a clear map of where the shortcuts work, where they break, and how to build a system that respects those boundaries to deliver both speed and reliability.

Technical Summary: Nexus

Problem Statement

Agentic Large Language Models (LLMs) utilizing the Model Context Protocol (MCP) face a significant performance bottleneck as tool registries scale. Each tool is described by a verbose JSON schema that must be re-encoded (prefilled) every turn. Since self-attention complexity scales quadratically ( $O(N^2)$ ) with sequence length, the accumulation of these schemas dominates the prompt, creating a "schema-prefill wall" that severely degrades Time-to-First-Token (TTFT) and eventually causes context window overflow.

While prior systems optimize KV-cache management by reusing blocks in-place (e.g., PagedAttention, RadixAttention), they do not relocate compiled blocks to new positions. A straightforward approach of compiling a schema once into a Key/Value (KV) block and transplanting it into a live context at inference time fails due to Rotary Position Embedding (RoPE) phase drift. RoPE injects position information by rotating query/key feature pairs; a block compiled for an anchor position $m_0$ carries incorrect absolute phase when placed at a runtime position $n_{past}$ , corrupting attention scores.

Methodology

Nexus is a user-space KV-cache serving prototype designed for Apple-silicon Unified Memory Architecture (UMA). It addresses the prefill wall through two primary mechanisms: retrieval-decoupled routing and depth-adaptive KV-cache splicing.

1. Retrieval-Decoupled Routing

To avoid the cost of re-prefilling schemas for every turn, Nexus decouples tool selection from the main context generation:

Semantic Lookaside Buffer (SLB): Tool embeddings (generated via nomic-embed-text-v1.5) are quantized to INT8 and stored in a dense buffer. Routing is performed via a branchless SIMD dot-product scan.
Calibrated Margin Gate: The top candidates are evaluated by a margin gate. If the score margin between the top two tools exceeds a calibrated threshold ( $\tau = 0.0136$ ), the decision is auto-routed. Otherwise, a fine-tuned cross-encoder reranks the top candidates.
Compressed Intermediate Representation (IR): Once a tool is selected, arguments are generated over a compressed textual signature (median 19 tokens) rather than the full spliced KV cache. This keeps the main-context prompt small and avoids relying on spliced KV for argument generation.

2. Depth-Adaptive KV Splicing and Repair

For the schemas that are transplanted into the live context, Nexus employs a "fail-closed" strategy to manage RoPE drift:

Anchored Splice: For context depths ( $n_{past}$ ) within a conservative threshold ( $P = 256$ tokens), the system performs a direct physical splice of the compiled KV block. The block is re-anchored (re-rotated) to match the runtime position, ensuring output-exactness (DKL $\approx 0$ ).
Depth-Adaptive Recompute: Beyond the 256-token threshold, the system initiates a repair process. It re-decodes a trailing suffix of the spliced schema. The fraction of re-decoded tokens, $R(n_{past})$ , scales linearly with depth, reaching 100% (a full re-prefill) at deeper contexts. This ensures a "never-regress" guarantee: the output fidelity (top-1 agreement and distributional identity) is never worse than a full text prefill, even if latency temporarily dips.
Systems Implementation: The system utilizes zero-copy memory mapping (mmap) for compiled blocks, transposed-V splicing for soft-capped attention models (e.g., Gemma-2), and cache-line hardening to ensure deterministic, bit-stable outputs under concurrency.

Key Contributions and Negative Results

The paper explicitly distinguishes between positive contributions and "bounding negative results" that define the system's architectural limits.

Positive Contributions

Retrieval-Decoupled Routing: A depth-independent routing mechanism that maintains ~89% accuracy even as the registry scales to 250 tools, where a "concatenate-all-schemas" baseline fails due to context overflow. It achieves a 1.66× speedup in reaching the first argument token with an ~80% reduction in main-context tokens.
Depth-Adaptive Recompute: A bounded repair mechanism that guarantees output fidelity (never-regress) by escalating re-decoding as context depth increases, converging gracefully to prefill parity.
UMA Systems Mechanisms: Implementation of transposed-V splicing for soft-capped models and cache-line hardened allocators for local execution.

Negative Results (Design Boundaries)

Off-Anchor RoPE Fidelity Boundary: The paper demonstrates that while a bare splice (without re-decoding) maintains top-1 agreement up to 2048 tokens, it introduces a residual distributional divergence ( $\sim 10^{-2}$ nats). This necessitates the repair mechanism to guarantee bit-identical output under any decoding parameters.
Failure of Reference-Free Drift Gating: An attempt to use a cheap, reference-free proxy (per-head K-variance) to predict when a deep splice would fail was unsuccessful. The proxy failed to rank-correlate with actual drift (Spearman $\rho = 0.193$ ), justifying the use of a deterministic, depth-adaptive repair curve instead.

Results

Measurements were conducted on an Apple M4 Max (64 GB UMA) using Qwen2.5-14B-Instruct (Q4_K_M).

Latency: At moderate depths ( $n_{past} = 256$ ), Nexus achieves a 1.63× to 1.73× TTFT speedup compared to full prefill. As depth increases and the recompute fraction $R(n_{past})$ approaches 100%, the speedup narrows to parity (1.0×), but the system never regresses in fidelity.
Routing Accuracy: Routing accuracy remains stable (~89%) as the tool registry scales from 10 to 250 tools. In contrast, a baseline that concatenates all schemas overflows the context window at $N \ge 50$ .
Argument Fidelity: On routed cases, the system fills 100% of query-specified arguments correctly with a 100% JSON validity rate. End-to-end argument accuracy (including routing errors) is 80%.
Micro-costs: The INT8 SLB scan over 250 tools completes in 17.6 $\mu$ s (including FFI overhead), demonstrating that routing overhead is negligible compared to prefill costs.

Significance and Scope

The paper positions Nexus not as a universal solution for all LLM serving, but as a characterization of the trade-offs between KV-cache transplantation and RoPE phase drift.

Qualitative Generalization: The paper claims that the qualitative boundaries—specifically that off-anchor relocation causes drift, scattered recompute cannot cheaply repair it, and reference-free gating fails to predict it—are properties of RoPE and hold model-agnostically.
Quantitative Specificity: The quantitative envelope (the 256-token threshold, specific recompute curve constants, and divergence magnitudes) is calibrated to a single model tuple (Qwen2.5-14B) and hardware (Apple M4 Max).
Scope Limitations: The system is designed for local, UMA-resident execution. It falls back entirely to text prefill for cloud APIs (which do not expose physical cache addresses) and discrete-GPU back-ends (where transposed strided transfers are pathological).

The author concludes that the most durable contribution of the work is the measured envelope and the negative results, which define the hard limits of schema splicing and justify the architectural choice to decouple routing from the splice path entirely.