🤖 machine learning

Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving

Irminsul introduces a native position-independent caching system for Agentic LLMs that leverages Multi-Head Latent Attention's architectural structure to enable content-addressed key-value caching, achieving up to 83% prompt token recovery and 63% prefill energy savings by overcoming the cache invalidation issues inherent in traditional prefix-matching approaches.

Original authors: Bole Ma, Jan Eitzinger, Harald Köstler

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bole Ma, Jan Eitzinger, Harald Köstler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a very busy, high-speed library where a librarian (the AI) has to read a massive stack of documents to answer a question. To be fast, the librarian keeps a "cheat sheet" (a cache) of the most recent pages they've read so they don't have to re-read them from scratch every time.

The Problem: The "Where" vs. The "What"

In the old way of doing things (called Prefix Caching), the librarian's cheat sheet was organized strictly by location.

"Page 1 is the System Prompt."
"Page 2 is Document A."
"Page 3 is Document B."

If the librarian asks a new question and the order changes slightly—say, "Page 1 is still the System Prompt, but now Document A is on Page 5"—the old cheat sheet breaks. The librarian thinks, "Oh, Page 2 is different, so I must throw away the whole cheat sheet and re-read everything from the beginning."

In the world of Agentic AI (AI that uses tools, searches the web, and talks to other AIs), this happens constantly. The AI might pull up the same document, but because it's inserting it into a different part of the conversation, the "location" changes. The old system sees this as a brand new, unreadable mess, forcing the AI to waste time and energy re-reading things it already knows.

The Solution: Irminsul (The "Content" Librarian)

The paper introduces a new system called Irminsul. Instead of organizing the cheat sheet by where a page sits, Irminsul organizes it by what the page actually says.

Think of it like this:

Old Way: "I only remember the book if it's on the 3rd shelf."
Irminsul: "I remember the book because it's a copy of Harry Potter, no matter which shelf it's on."

Irminsul breaks the conversation into chunks based on the actual text (content). If the AI sees "Document A" again, even if it's in a different spot, Irminsul says, "I've seen this exact text before! I can reuse my notes."

The Secret Sauce: The "Position" Adjustment

You might ask: "If the text is the same, but the position is different, won't the AI get confused?"

Yes, usually. In AI, the "position" of a word matters a lot (it's like knowing if a word is at the start or end of a sentence).

The Old Problem: To fix the position, the old systems had to rewrite the entire cheat sheet for every single word. It was like rewriting a whole book just to change the page number. This was slow and expensive.
The Irminsul Trick: The paper focuses on a specific type of AI architecture called MLA (Multi-Head Latent Attention). This architecture is special because it splits the "notes" into two parts:
1. The Content (90%): The actual meaning of the words. This part is identical regardless of position.
2. The Position (10%): A tiny tag that says "I am here."

Irminsul realizes it only needs to adjust that tiny 10% tag. It uses a clever mathematical "twist" (called a $\delta$ -rotation) to instantly update the position tag without rewriting the whole book. It's like taking a photo of a document and just sliding it to a new spot on the desk, rather than taking a new photo of the whole room.

The Results

The paper tested this on three real-world AI systems (DeepSeek, Kimi, and JoyAI) that act like agents.

Recovery: On complex tasks where the AI moves documents around, Irminsul was able to reuse about 77% to 83% of the work that the old system threw away.
Energy: Because it doesn't have to re-read the text, it saves about 63% of the energy needed to process those repeated parts.
Accuracy: The AI answers just as correctly as before; it doesn't get confused by the new method.

The "First Page" Rule

There is one small catch. The very first few words of any conversation act like a "gravity anchor" for the AI's attention. The paper found that if you try to reuse a chunk that starts right at the beginning of a conversation, the AI gets confused.

The Fix: Irminsul simply re-reads the very first chunk of text (the first 32 words) every time, but for everything after that, it uses the smart "content-based" reuse. This tiny cost ensures the rest of the system works perfectly.

Summary

Irminsul is a smarter way for AI to remember things. Instead of saying, "I remember this because it was in slot #5," it says, "I remember this because it's the same story." By realizing that the "story" (content) is more important than the "slot" (position), and by using a special trick to fix the position tag quickly, it makes AI agents faster, cheaper to run, and much better at handling complex, shifting conversations.

Technical Summary: Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving

1. Problem Statement

Current production LLM serving relies heavily on prefix caching (e.g., SGLang's RadixAttention, vLLM's PagedAttention), which caches Key-Value (KV) states based on exact token-sequence prefixes. While effective for standard conversational turn-taking, this mechanism fails catastrophically in agentic workloads.

In agentic systems (e.g., tool-use, retrieval-augmented generation, multi-agent orchestration), prompts are dynamically assembled from static system instructions, retrieved documents, and tool outputs. Even if the content of a document or tool schema remains identical across turns, its position within the prompt shifts due to:

Dynamic insertion of retrieved documents.
Re-ordering of tool outputs.
Variable-length system prompts or metadata headers.

Because standard prefix caches require byte-identical alignment from position 0, a single byte shift in the prompt structure invalidates the entire cache for downstream tokens. Operators report severe regressions, including cache hit rates dropping from 100% to 35% and Time-To-First-Token (TTFT) spikes of 10–16 seconds on unchanged content.

Prior Position-Independent Caching (PIC) systems (e.g., CacheBlend, EPIC) attempt to solve this for Grouped-Query Attention (GQA) architectures by correcting the full $d_K$ -dimensional Key tensor via inverse rotation. However, this incurs a significant architectural cost: the correction must be applied to the entire KV row per token, making it computationally expensive and often negating the benefits of caching.

2. Methodology: The Irminsul Approach

Irminsul proposes a content-addressed caching system specifically designed for Multi-Head Latent Attention (MLA) architectures (used in DeepSeek-V2/V3/R1, Kimi-K2, GLM-5, Mistral Large 3).

2.1 Structural Insight: MLA Decomposition

The core enabler for Irminsul is the structural decomposition of the KV pair in MLA models. Unlike GQA, where position is inseparably fused into the full Key tensor, MLA factors each token's KV into:

$cKV$ (Content Latent): A 512-dimensional, position-free latent vector. This component is reusable verbatim at any absolute position.
$k_r$ (Rotary Key): A small, 64-dimensional vector carrying the positional information (RoPE).

Irminsul exploits the fact that $cKV$ constitutes ~~89% of the KV row and requires zero correction when reused. Only the $k_r$ slice (~~11%) requires adjustment, which can be corrected in closed form via a single $\delta$ -rotation (where $\delta$ is the position shift).

2.2 System Design

Irminsul extends SGLang's radix cache with the following mechanisms:

Content-Hash Keying with CDC: Instead of hashing fixed blocks, Irminsul uses Content-Defined Chunking (CDC) based on a rolling Gear-hash over 64-token windows. Chunks are emitted when the hash falls below a threshold, ensuring chunk boundaries align with content rather than arbitrary byte offsets. Each chunk is fingerprinted using xxHash64.
Registry Lookup: The cache registry maps content hashes to stored $(cKV, k_r, base\_pos)$ .
$\delta$ -Rotation Rule: Upon a cache hit, the system calculates the position shift $\delta = p_{target} - p_{source}$ . It retrieves the stored $cKV$ and applies a rotation matrix $R(\delta)$ to the stored $k_r$ to generate the correct positional key for the new location. This operation is fused into the FlashMLA kernel, incurring negligible latency.
First-Chunk Carve-Out: To handle the "attention sink" phenomenon (where the model attends heavily to the sequence start), Irminsul pre-fills the first 32 tokens of any chunk via standard prefill if the chunk starts at absolute position 0. This avoids the need for boundary re-computation on subsequent chunks, as the sink is sequence-start-local, not chunk-start-local.
Hybrid Strategies (S1/S2):
- S2 (Prevent via Cooperation): In controlled environments, the prompt assembler inserts a 64-token boundary marker around shared regions to force deterministic CDC boundaries.
- S1 (Detect and Recover): For uncontrolled traffic, a fallback mechanism subdivides CDC misses into fixed 128-token windows to recover partial reuse.

3. Key Contributions

Irminsul System: The first MLA-native PIC system. It achieves high-precision $\delta$ -rotation with a relative L2 error of $4.7 \times 10^{-3}$ in bf16, which is an order of magnitude tighter than standard FP8 KV quantization.
Feasibility Proof (ROC Analysis): The paper provides a threshold-free ROC study demonstrating that MLA's $cKV$ component is highly position-invariant (AUC $\approx$ 0.77), whereas raw K vectors in GQA/MHA are position-entangled (AUC < 0.5). This validates that content-addressed reuse is structurally sound for MLA but not for pure softmax architectures without correction.
Energy Efficiency Study: A cross-architecture analysis showing that while softmax-based attention (GQA, MLA, MHA) saves 63–86% of prefill energy per cache hit, hybrid SSM/linear architectures (Mamba2, GDN, KDA) save ~0% because their recurrent state is not token-indexed, making "cache hits" undefined at the recurrent layer.
Recovery Metrics: Evaluation on three native MLA-MoE deployments (DeepSeek-V2-Lite, Kimi Moonlight-16B, JoyAI-Flash) showing that Irminsul recovers up to ~83% of prompt tokens above exact-prefix matching on agentic traffic, compared to near-zero recovery for standard prefix caches on shifted content.

4. Results

Cache Recovery: On agentic traffic with "early variation" (e.g., shifted agent metadata headers), Irminsul recovers 77.2% (DeepSeek-V2-Lite) and 82.7% (JoyAI-48B) of tokens that would otherwise be recomputed. Standard prefix caching captures only ~1.5% in these scenarios.
Energy Savings: Per cache hit, Irminsul delivers 63% prefill energy savings on MLA models (compared to 86% on GQA). This confirms that the primary benefit of PIC is realized in softmax-based architectures.
Output Consistency: Across four datasets (QA, GovReport, NIAH) and three models, Irminsul's greedy trajectories match or exceed naive reuse. On the "Needle In A Haystack" (NIAH) task with JoyAI-48B, PIC matches the full prefill recall exactly (0.820) while reducing KL divergence by 22% compared to naive reuse.
Latency: The system operates within production latency budgets. The $\delta$ -rotation is fused into the HBM-to-SRAM load path, avoiding extra HBM bandwidth usage.

5. Significance and Claims

The paper argues that content-addressed caching should be a first-class primitive in the LLM serving stack, rather than a retrofit over prefix matching.

Structural vs. Algorithmic: The paper posits that the inability of prior PIC systems to fully solve the problem for GQA is a structural limitation (fusing position into the full Key tensor), not just an engineering shortcoming. MLA's factorization of position ( $k_r$ ) from content ($cKV$) is the key that makes efficient, low-overhead PIC possible.
Agentic Workloads: As agentic systems become dominant, the "token re-encoding" of shared context (documents, schemas) represents a massive inefficiency. Irminsul reframes this redundancy not as a compute problem, but as a memory problem, allowing the serving stack to bill for memory reuse rather than recomputation.
Scope: The authors explicitly limit their safety claims to MLA architectures (specifically the absorbed-matmul form used in production). They note that while GQA can theoretically support PIC, the correction cost is high. They also note that hybrid SSM models (like Mamba2) do not benefit from PIC due to their recurrent state structure.
Future Work: The paper acknowledges that DeepSeek-V4's "Heavily Compressed Attention" (HCA) aggregates tokens, which dissolves the per-token identity required for Irminsul's chunking. Adapting the chunking primitive to HCA's aggregation schedule is identified as the immediate next step.

In summary, Irminsul demonstrates that by leveraging the specific algebraic properties of MLA, it is possible to cache content regardless of its position in the prompt, recovering significant compute and energy costs in agentic LLM serving without compromising output quality.