💬 NLP

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

SeKV is a resolution-adaptive KV cache compression method that organizes context into semantic spans stored across a GPU-CPU hierarchy, enabling on-demand token-level reconstruction via a trained zoom-in mechanism to significantly reduce GPU memory usage while improving long-context inference performance without fine-tuning the base model.

Original authors: Amirhossein Abaskohi, Giuseppe Carenini, Peter West, Yuhang He

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amirhossein Abaskohi, Giuseppe Carenini, Peter West, Yuhang He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Too Much Stuff" Bottleneck

Imagine you are trying to read a massive 128,000-page novel to answer a single question. To do this, your brain (the AI model) needs to keep the whole book open in front of you so you can flip back and find the answer.

In AI terms, this "open book" is called the KV Cache.

The Issue: As the book gets longer, the space required to keep it open grows linearly. Eventually, your brain (the computer's GPU memory) runs out of space.
The Current Fix (and why it fails): To save space, current methods try to throw away pages they think are boring.
- Analogy: Imagine you are reading a mystery novel. You decide to throw away the middle 50 pages because they seem like "just descriptions of the weather." But later, you realize the killer's alibi was hidden in a weather report on page 4,000. Because you threw those pages away, you can never find the answer. You have to guess (hallucinate).

The Solution: SEKV (The "Smart Library" System)

The authors propose SEKV, a new way to manage this memory. Instead of throwing pages away, SEKV organizes the book into chapters and uses a two-tier storage system (like a desk and a basement).

Here is how it works, step-by-step:

1. Smart Chaptering (Entropy-Guided Segmentation)

Instead of cutting the book into random chunks (like "every 100 words"), SEKV looks for natural breaks in the story.

How it works: It uses a signal called "surprisal." If a word is unexpected or marks a big change in topic (like a new character entering or a plot twist), that's a chapter break.
The Analogy: Think of a librarian who knows exactly where the plot twists happen. They don't just cut the book in half; they group the story into logical "scenes."

2. The Two-Tier Memory (GPU vs. CPU)

SEKV splits the storage between a fast, expensive surface (the GPU) and a slower, massive storage area (the CPU).

The Desk (GPU): This is your immediate workspace.
- What's here: The very beginning of the book, the very end (what you just read), and one "Anchor Token" from every chapter.
- The Anchor: This is a single, high-quality sentence from the start of a chapter that summarizes the whole scene. It's like a bookmark that says, "This chapter is about the EU negotiator demanding a 40% emissions cut."
- The Summary: Each chapter also has a tiny, compressed "summary card" on the desk to help you decide if you need to look deeper.
The Basement (CPU): This is where the rest of the book lives.
- What's here: The full, detailed text of every chapter, but compressed using a mathematical trick called SVD (think of it as a highly efficient zip file).
- The Magic: Nothing is deleted. The full details are safe in the basement, just waiting to be called up.

3. The "Zoom-In" Mechanism

This is the most important part. When the AI is answering a question, it doesn't just guess. It follows a process:

Scan the Desk: The AI looks at the "Anchor Tokens" and "Summary Cards" on the GPU.
Spot the Match: If the question is about the "EU negotiator," the AI sees the anchor token for that chapter and realizes, "I need the details from this specific chapter."
The Zoom-In: The AI triggers a "Zoom-In" command. It goes to the basement (CPU), fetches the compressed "zip file" for that specific chapter, and unzips it back into full detail on the desk.
Answer: Now the AI has the full, high-resolution text of that specific chapter to find the exact answer ("40% by 2035").

Why This is Better Than Old Methods

Old Method (Token Eviction): "I think this page is boring, so I burn it." If you're wrong, the information is gone forever.
SEKV: "I think this page might be boring, so I put it in a compressed box in the basement. But if you ask about it later, I can instantly pull it out and open it."

The Results

The paper tested this system on four different "exams" (benchmarks) involving long documents and complex reasoning.

Accuracy: SEKV was 5.9% more accurate than the best previous methods at finding answers in long texts.
Memory: It used 53% less GPU memory than keeping the whole book open, while still being able to find the answer.
Efficiency: It didn't slow things down much because it only "zooms in" on the specific parts of the book that are actually needed for the question.

Summary

SEKV is like having a librarian who never throws away a page. Instead, they organize the library into logical chapters, keep the most important summaries on your desk, and store the rest in a basement. When you need a specific detail, they instantly retrieve and expand just that one chapter, saving you space without losing any information.

Technical Summary: SEKV – Resolution-Adaptive KV Cache with Hierarchical Semantic Memory

Problem Statement

Large Language Models (LLMs) are increasingly deployed in scenarios requiring long-context reasoning (e.g., document analysis, multi-turn dialogue, agentic planning). However, the Key-Value (KV) cache, which stores intermediate states to accelerate decoding, grows linearly with sequence length. Retaining a full KV cache for contexts exceeding 128K tokens becomes prohibitively expensive in terms of GPU memory, often rendering long-context inference infeasible without compression.

Existing compression strategies face a fundamental trade-off between efficiency and fidelity:

Token Eviction: Methods that permanently discard tokens based on attention statistics (e.g., H2O, SnapKV) risk fragmenting coherent evidence or irretrievably losing information that becomes critical only later in the generation process.
Static Semantic Grouping: Methods that compress tokens into semantic chunks or sentences at prefill time (e.g., ChunkKV, SentenceKV) fix the compression resolution early. They lack the ability to recover fine-grained, token-level details from a compressed span once that span becomes relevant during decoding.

The core challenge is to design a memory organization that preserves access to relevant evidence under tight memory constraints without irreversible information loss, allowing the model to dynamically adjust the resolution of stored context based on query relevance.

Methodology: SEKV

The authors propose SEKV (Semantic KV Cache), a resolution-adaptive semantic KV cache that organizes context into a hierarchical semantic memory spanning GPU and CPU. SEKV avoids irreversible token eviction by storing context at variable resolutions and dynamically "zooming in" to the appropriate level of detail during decoding.

1. Entropy-Guided Span Segmentation

Instead of fixed-size chunking, SEKV segments the input context into semantically coherent spans using token surprisal as a zero-cost boundary signal.

Mechanism: During the prefill forward pass, the surprisal $H_t = -\log p(x_t | x_{<t})$ is computed for each token. High-surprisal tokens, which often mark semantic transitions (topic shifts, entity introductions), are identified as span boundaries.
Anchor Tokens: These high-surprisal boundary tokens are retained at full resolution on the GPU as "anchor tokens." They carry concentrated semantic information essential for later routing and reconstruction.
Spans: Tokens between two neighboring anchors form a semantic span $S$ .

2. Dual-Resolution Span Representation & Memory Hierarchy

Each semantic span is represented simultaneously at two resolutions across a GPU–CPU hierarchy:

GPU (Coarse Routing): Each span is represented by a lightweight summary vector (a normalized, surprisal-weighted average of projected key vectors). This summary resides on the GPU and is used for efficient, coarse-grained routing to determine which spans are relevant to the current query.
CPU (Fine-Grained Reconstruction): The full token-level structure of the span is compactly encoded on the CPU using a low-rank SVD basis. A lightweight predictor assigns per-component soft gates to the singular values, allowing the system to adaptively retain a rank budget $r_S$ for each span based on its information density. This enables faithful, on-demand reconstruction of token-level KV pairs without storing the full matrix on the GPU.

3. Trained Zoom-In Mechanism

During decoding, SEKV employs a learned mechanism to dynamically select which spans to expand:

Routing: The current query is projected into the same routing space as the span summaries. A sigmoid relevance gate calculates the probability of a span being relevant.
Expansion: A learned, per-head, per-layer threshold determines whether a span is expanded. If selected, the system asynchronously fetches the corresponding low-rank SVD basis from the CPU, reconstructs the token-level KV pairs, and integrates them into the attention computation.
Mixed-Resolution Attention: The attention mechanism operates on a mixed-resolution memory: anchor tokens contribute full KV pairs, non-expanded spans contribute their coarse summary vectors (with a size correction), and expanded spans contribute reconstructed token-level pairs.
Training: The base LLM remains frozen. Only the lightweight routing projections, zoom-in thresholds, and the rank-gate predictor are trained using a combination of distillation loss (aligning with a full-KV teacher), zoom supervision, reconstruction loss, and a budget penalty.

Key Contributions

SEKV Architecture: A resolution-adaptive semantic KV cache that stores context at multiple resolutions and dynamically adjusts cache detail based on query relevance, avoiding irreversible token eviction.
Entropy-Guided Segmentation: A method using token surprisal to identify coherent semantic spans and preserve high-surprisal anchor tokens at full resolution for reliable retrieval.
Dual-Representation Structure: A novel KV structure combining GPU-resident span summaries for coarse routing, CPU-resident low-rank bases for approximate token-level reconstruction, and a trained zoom-in mechanism that selectively expands only query-relevant spans.

Experimental Results

SEKV was evaluated across four benchmarks (LongBench, RULER, InfiniteBench, Needle-in-a-Haystack) and five backbone models (including Llama-3 variants, Mistral, and Qwen).

Performance: SEKV outperforms the strongest semantic compression baseline (SentenceKV) by an average of 5.9% across all benchmarks and models. It achieves the best results in 20 out of 20 benchmark-model comparisons.
Retrieval Fidelity: On the Needle-in-a-Haystack (NIAH) benchmark, SEKV shows superior retrieval stability across varying context lengths and needle depths compared to token-eviction and static semantic methods.
Memory Efficiency: At a 128K context length, SEKV reduces GPU memory consumption by 53.3% compared to full KV caching while maintaining performance close to the uncompressed baseline.
Efficiency: SEKV achieves competitive latency and throughput, often outperforming full KV caching in latency due to reduced memory bandwidth pressure, while significantly outperforming aggressive eviction methods in accuracy.
Parameter Efficiency: The method adds fewer than 0.05% trainable parameters (approx. 4.3M parameters for Llama-3-8B) while keeping the base model frozen.

Significance and Claims

The paper argues that efficient long-context inference requires a shift from static compression to dynamic resolution recovery. The authors claim that while static compression preserves coarse context, it fails to retrieve fine-grained evidence when it becomes relevant late in the generation process. SEKV addresses this by organizing context into entropy-guided semantic spans and utilizing a hierarchical memory system that allows the model to "zoom in" on relevant information on demand.

The significance of this work lies in demonstrating that:

Irreversible eviction is not necessary: High-fidelity long-context inference can be achieved by offloading recoverable low-rank representations to CPU rather than discarding tokens.
Adaptive resolution is critical: Allocating fine-grained resolution only where needed (via the zoom-in mechanism) allows models to balance memory constraints with retrieval fidelity more effectively than static policies.
Semantic structure aids compression: Using entropy-guided segmentation creates more compressible units than arbitrary token blocks, enabling efficient low-rank approximation.

The authors acknowledge limitations, noting that effectiveness depends on the quality of semantic span construction (which may struggle with highly fragmented discourse) and that the approach introduces a dependency on CPU-GPU bandwidth, which could impact latency in adversarial query scenarios. However, they conclude that SEKV provides a robust framework for scalable, high-fidelity long-context LLM inference.