🤖 AI

LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs

The paper introduces LinearKV, a training-free framework that enables position-independent caching in hybrid LLMs by demonstrating that initializing linear recurrence layers with a single cached state is both more effective and efficient than the algebraically exact composition of all cached states used in concurrent methods.

Original authors: Yirui Liu, Ruoling Qi, Longwen Wang, Xuaner Wu, Jian Chen, Yuxin Jin, Jiawei Shao, Xuelong Li

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yirui Liu, Ruoling Qi, Longwen Wang, Xuaner Wu, Jian Chen, Yuxin Jin, Jiawei Shao, Xuelong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, super-smart library where a robot librarian reads books to answer your questions. The problem is, the library is growing so fast that reading a whole book from the very first page every time you ask a question takes forever. To speed things up, the librarians started a clever trick: they memorize chunks of books they've already read. If you ask about a story they've seen before, they just pull out their notes instead of re-reading the whole thing. This is called "caching." But there's a catch: usually, they can only use those notes if the story starts exactly the same way as before. If you change the beginning, the notes become useless.

Recently, scientists invented a new way to make these libraries even faster by mixing two types of reading styles. One style is like a traditional librarian who remembers every single word (Full Attention), while the other is like a super-efficient robot that only remembers a single "summary state" of the story so far (Linear/Recurrent). This hybrid approach is great, but it broke the old caching trick. The old trick relied on stitching together pages of notes, but the new robot style doesn't have pages to stitch; it only has a single summary state. So, the big question became: Can we still use the "notes from anywhere" trick with these new hybrid robots, or do we have to start from scratch every time?

This paper, titled LINEARKV, answers that question with a surprising twist. The researchers found that you can use the "notes from anywhere" trick with these hybrid models, but the way you combine the notes matters more than you'd think. They discovered that the most logical, mathematically perfect way to combine the notes from different chunks of a story actually makes the robot librarian confused and gives terrible answers. Instead, the best strategy is shockingly simple: just pick the notes from the very last chunk you found and use that as your starting point.

Here is how they figured it out. When the hybrid robot reads a chunk of text, it compresses everything it learned into a tiny "state" (a summary). If you have three chunks of text cached, you have three of these summaries. The "perfect" math way to combine them is to try to reconstruct exactly what the robot's brain would look like if it had read all three chunks in order from the start. The authors call this "exact composition." It sounds like the right thing to do, like trying to perfectly reassemble a puzzle. However, when they tested this on a specific type of hybrid model called Mamba-2, it completely failed. The robot got so confused that it only recovered about 46.6% of the quality of a fresh read.

On the other hand, the "single-summary" method—just taking the summary from the last chunk and ignoring the rest—worked amazingly well. It lifted the quality to 86.8% of a fresh read. It turns out that trying to mathematically glue the summaries together introduces errors that pile up and break the robot's logic. By just using the most recent summary, the robot avoids these errors and stays on track. Interestingly, on the other type of hybrid model they tested (called GDN), both the "perfect math" way and the "single-summary" way worked about the same, recovering up to 92% of the quality.

The researchers also checked how fast this was. Using the "single-summary" method was not only more accurate for the Mamba-2 model but also faster. It cut the time to get the first answer down to 0.46 times the time it takes to read the whole thing from scratch, whereas the "perfect math" way was slightly slower and still gave bad answers.

In short, the paper shows that for these new hybrid AI models, you don't need to do complex math to reuse old memories. In fact, doing the complex math can hurt you. The best approach is to keep it simple: grab the memory from the last piece of the puzzle you found, and let the AI fill in the gaps. This method works across different types of long-document tasks, from answering questions about history to tracking variables in a story, proving that sometimes the simplest solution is the smartest one.

Technical Summary: LINEARKV

Problem Statement
Large Language Model (LLM) serving faces significant latency and throughput bottlenecks due to the growing cost of prefilling long contexts. Position-Independent Caching (PIC) mitigates this by storing and reusing independently prefilled token chunks regardless of their position in the prompt. However, existing PIC methods are designed for full-attention architectures, relying on token-indexed Key-Value (KV) caches that can be concatenated and selectively repaired. Hybrid LLMs, which interleave full-attention (FA) layers with recurrent layers (e.g., Mamba-2, Gated DeltaNet), break these primitives. Recurrent layers compress context into a fixed-size state rather than maintaining a token-indexed cache. This creates two challenges: (1) It is unclear how to initialize and repair recurrent states from disjoint cached chunks without token-indexed entries to concatenate, and (2) It is unknown whether a unified PIC strategy can generalize across different hybrid architectures (e.g., scalar-decay Mamba-2 vs. dense-transition Gated DeltaNet).

Methodology: LinearKV
The authors propose LinearKV, a training-free framework for position-independent caching in hybrid LLMs. The core innovation is a decoupled initialization strategy applied after matching cached chunks:

Decoupled Initialization:
- Full-Attention Layers: These layers proceed as in standard PIC, concatenating cached KV entries from matched chunks along the token axis.
- Recurrent (Linear) Layers: Instead of attempting to concatenate states (which is impossible as they are fixed-size summaries), the framework maps $K$ matched local states $\{S^{\text{local}}_1, \dots, S^{\text{local}}_K\}$ into a single initial state $S_{\text{init}}$ via a function $f$ .
- Selector Reuse: This decoupling allows LinearKV to reuse existing PIC token-selection algorithms (e.g., CacheBlend, EPIC, ProphetKV) unchanged. The selector determines a set of token positions $U$ to recompute. The system then repairs the cache by overwriting FA KV entries at $U$ and advancing the recurrent state from $S_{\text{init}}$ through the tokens in $U$ in context order.
The Linear-State Initializer ( $f$ ):
The paper investigates two choices for $f$ :
- Exact Composition: An algebraically principled approach (adopted by concurrent work HYPIC) that composes all $K$ cached states using the transition operators of the chunks to reconstruct the exact full-prefix state.
- Single-Block Initialization (LinearKV's default): A simpler approach where $f$ selects a single cached state (typically the last matched chunk, $S^{\text{local}}_K$ ) as the initializer, discarding the others.

Key Findings and Contributions
The paper makes three primary contributions, supported by empirical evidence across three hybrid models (Granite/Mamba-2, OLMo/GDN, Qwen/GDN) and three PIC selectors:

Formalization of Hybrid PIC: The authors formalize hybrid PIC as a decoupled framework where the only architecture-specific operation is the linear-state initialization. This allows existing full-attention PIC selectors to be plugged in without modification.
Fragility of Exact Composition: Contrary to algebraic intuition, the paper demonstrates that exact composition is unnecessary and architecture-fragile.
- On GDN models (OLMo, Qwen), exact composition and single-block initialization perform equivalently, both recovering up to 92% of full quality.
- On Mamba-2 models (Granite), exact composition collapses. Due to the scalar-decay nature of Mamba-2, errors from independently prefilled chunks compound through depth, leading to a constructed initial state with high relative error. Consequently, exact composition recovers only ~46.6% of full quality under the EPIC selector, whereas the single-block initializer recovers 86.8%.
- The authors identify that the failure stems from the aggregation of $K$ independently conditioned deep operators, not the architecture itself, as the single-block approach avoids this error accumulation.
Efficiency Gains: The single-block initializer is computationally cheaper. It avoids the online cost of composing dense transition matrices required by exact composition. This reduces the Time-to-First-Token (TTFT) by an additional 5–17% compared to exact composition, while maintaining or improving accuracy.

Experimental Results

Benchmarks: Evaluated on LongBench QA, RULER (8K–32K), and LongBench Summarization.
Performance:
- Granite (Mamba-2): LinearKV (single-block) significantly outperforms exact composition across all selectors. For example, under EPIC at 20% recomputation budget, LinearKV achieves 86.8% of full quality versus 46.6% for exact composition.
- OLMo & Qwen (GDN): Both methods perform similarly, with LinearKV matching the performance of exact composition.
- Robustness: The superiority of the single-block initializer on Mamba-2 holds across varying recomputation ratios (0.03 to 0.4) and context lengths (8K to 32K).
Latency: LinearKV reduces TTFT to 0.46×–0.62× of full prefill time, outperforming exact composition in all measured scenarios.

Significance
The paper establishes that for hybrid LLMs, the "algebraically principled" approach of reconstructing the exact full-prefix state from cached chunks is not only computationally expensive but can be detrimental to model performance on specific architectures like Mamba-2. By demonstrating that a single cached state suffices as a robust initializer, LinearKV provides a simple, training-free, and architecture-agnostic solution that maximizes the benefits of position-independent caching while minimizing latency and avoiding the error compounding inherent in multi-source state composition. This finding challenges the assumption that more complex state composition is necessary for effective caching in recurrent architectures.

Technical Summary: LINEARKV

More like this