💬 NLP

Latent-Condensed Transformer for Efficient Long Context Modeling

This paper introduces Latent-Condensed Attention (LCA), an architecture-agnostic method that jointly optimizes computational efficiency and KV cache reduction for long-context modeling by directly condensing context within the latent space of Multi-head Latent Attention (MLA) through query-aware pooling and anchor selection, achieving significant speedups and memory savings without compromising performance.

Original authors: Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang, Mingkui Tan

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang, Mingkui Tan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to read a massive, 100,000-page encyclopedia to answer a single question.

The Problem: The "Overwhelmed Librarian"
Current AI models (Large Language Models) act like a brilliant but overwhelmed librarian. When you ask a question, the librarian tries to remember every single word they've ever read.

The Memory Issue: As the book gets longer, the librarian's "sticky notes" (called the KV Cache) grow linearly. For a 100,000-page book, they need a sticky note for every single page. This fills up their desk (memory) instantly.
The Speed Issue: To find the answer, the librarian has to compare your question against every single page they've read. If the book is 100,000 pages long, they have to do 100,000 x 100,000 comparisons. This is like trying to find a needle in a haystack by checking every single straw against every other straw. It's incredibly slow.

The Current "Fixes" (And Why They Fall Short)
Scientists have tried two main tricks to help:

Trick A (MLA): Instead of writing a full sticky note for every page, they write a tiny, compressed summary. This saves desk space, but the librarian still has to read through all those tiny summaries to find the answer. It's still slow.
Trick B (Sparse Attention): The librarian decides to only read pages 1, 100, 500, and 999, ignoring the rest. This is fast, but they might miss the crucial page (say, page 450) that actually holds the answer.

The New Solution: "Latent-Condensed Attention" (LCA)
This paper introduces a new way to organize the librarian's desk. Instead of just summarizing or skipping pages, the librarian groups the book into chapters and creates a "Super-Chapter" for each.

Here is how LCA works, using a creative analogy:

1. The "Grouping" Strategy

Imagine the 100,000 pages are divided into groups of 16 pages each.

The Semantic Part (The "Story"): For the content of the story in those 16 pages, the librarian doesn't pick just one page. Instead, they blend the most important sentences from all 16 pages into a single, perfect "Super-Sentence."
- Analogy: If the 16 pages are about a cat, the Super-Sentence captures the essence of the cat perfectly, without needing to read all 16 pages individually.
The Positional Part (The "Address"): However, where the story happens matters. You can't just blend the addresses of 16 different houses into one. So, for the location, the librarian picks the single most important page in that group and keeps its exact address.
- Analogy: If the story is about a cat in a kitchen, the librarian keeps the exact address of the kitchen, ensuring the AI knows where the cat is.

2. The "Local Window" (The "Recent Memory")

The librarian knows that the very last few pages are usually the most important for the current question. So, they keep the last 1,000 pages in full, high-definition detail, untouched. They only start "compressing" the older, distant history.

3. The Result: A Super-Efficient Librarian

By doing this, the librarian transforms a 100,000-page book into:

A few thousand "Super-Sentences" (representing the old history).
A few thousand "Exact Addresses" (to keep track of where things are).
The last 1,000 pages in full detail.

Why is this a game-changer?

Speed: Instead of comparing your question against 100,000 pages, the librarian only compares it against a few thousand "Super-Groups." The paper shows this makes the AI 2.5 times faster at reading long documents.
Memory: The "sticky notes" (KV Cache) shrink by 90%. The librarian can now handle massive books without running out of desk space.
Accuracy: Because they blend the meaning of the whole group rather than just deleting pages, they don't lose important information. They also keep the exact location of the most important parts, so they don't get confused about where things happened.

The Bottom Line

Think of LCA as a smart filing system that doesn't just throw away old files or shrink them blindly. Instead, it creates smart summaries for the past while keeping exact coordinates for where things happened, and keeps the recent news in full color.

This allows AI to read entire libraries in the time it used to take to read a single chapter, without forgetting the details that matter.

1. Problem Statement

Large Language Models (LLMs) face two primary bottlenecks when processing long contexts:

Memory Overhead: The Key-Value (KV) cache grows linearly with sequence length, consuming significant GPU memory.
Computational Complexity: Self-attention mechanisms exhibit quadratic complexity ( $O(L^2)$ ) relative to context length, slowing down inference.

Existing Limitations:

Multi-head Latent Attention (MLA): Addresses memory by projecting tokens into a low-dimensional latent space, significantly reducing KV cache size. However, it still performs dense attention over all tokens, retaining the $O(L^2)$ computational bottleneck.
Sparse Attention: Reduces computation by skipping tokens or blocks. However, most sparse methods operate on reconstructed full-dimensional KV matrices. When applied to MLA, they must first decompress the latent vectors back to full dimensions to calculate attention scores, negating MLA's memory efficiency gains and failing to jointly optimize both memory and compute.

The Gap: There is a lack of methods that can perform efficient computation and memory reduction natively within the compressed latent space of MLA without reconstructing full-dimensional representations.

2. Methodology: Latent-Condensed Attention (LCA)

The authors propose Latent-Condensed Attention (LCA), a mechanism that directly condenses context within MLA's latent space. The core innovation is a dual-path condensation strategy that treats semantic and positional information differently, acknowledging their distinct functional properties.

A. Latent-Space Condensation

LCA partitions the history context (excluding a local window) into contiguous groups. For each group, it performs two distinct operations:

Semantic Condensation (Weighted Pooling):
- Target: The semantic latent vectors ( $C_{KV}$ ).
- Mechanism: Instead of discarding tokens, LCA aggregates them into a single representative vector using query-aware weighted pooling.
- Logic: It computes importance scores based on the similarity between a summary query (averaged from recent tokens) and the keys in the group. The semantic vectors are then combined via a weighted sum.
- Theoretical Basis: The paper proves (Proposition 1) that this weighted pooling minimizes the expected squared reconstruction error, making it the optimal condensation strategy for semantic content.
Positional Anchoring (Max Selection):
- Target: The positional keys ( $K_R$ ), which are encoded via Rotary Position Embedding (RoPE).
- Mechanism: Unlike semantic vectors, positional information is non-linear and cannot be averaged without distorting relative positions. LCA uses hard anchor selection: it selects the single token with the highest importance score in the group to represent the positional key for that entire group.
- Benefit: This preserves the precise geometric relationships required for RoPE without "blending" incompatible positional signals.

B. Local Window Preservation

To ensure fine-grained local context is not lost, LCA retains a local window ( $w$ ) of the most recent tokens in full fidelity (no condensation). This ensures that immediate context and the most relevant recent tokens are attended to with full precision.

C. Integration and Inference

Fused Context: During attention computation, the condensed representatives ( $K_{rep}, V_{rep}$ ) are concatenated with the full-fidelity local window ( $K_{loc}, V_{loc}$ ).
Decoding: During autoregressive generation, the cache stores only the condensed representatives and the local window. As new tokens are generated, the oldest group in the buffer is condensed into a new representative, keeping the cache size constant at $O(m + w)$ .
Architecture Agnostic: While designed for MLA, the strategy is generalizable to other attention mechanisms like Grouped-Query Attention (GQA).

3. Key Contributions

Novel Mechanism: Introduced LCA, the first method to perform structured context condensation natively within the latent space of MLA, avoiding the overhead of full-dimensional reconstruction.
Decoupled Strategy: Proposed a dual-path approach that adaptively aggregates semantic information (via weighted pooling) while preserving positional fidelity (via anchor selection), solving the signal-blending issue common in conventional pooling.
Theoretical Guarantees: Proved a length-independent error bound (Theorem 1), demonstrating that the approximation error depends only on per-token deviations and model dimensions, not on the total context length $L$ .
Zero Parameter Overhead: The method requires no additional learnable parameters and can be integrated into pretrained models via lightweight fine-tuning.

4. Experimental Results

The authors evaluated LCA on DeepSeek-V2-Lite (16B) and MiniCPM3-4B, comparing it against state-of-the-art methods like MInference, FlexPrefill, and KDA.

Performance (Accuracy):
- Long Context: On LongBench-E and RULER (up to 128K context), LCA maintains competitive performance, often matching or slightly exceeding the original MLA baseline. For example, at 128K context on RULER, LCA achieved 24.38 vs. MLA's 23.96.
- Short Context: On standard benchmarks (MMLU, GSM8K, MBPP), LCA performs nearly identically to the baseline, proving it does not degrade general capabilities.
- Comparison: Unlike sparse methods (MInference/FlexPrefill) which suffer significant accuracy drops (e.g., dropping to ~4.34 on RULER at 128K), LCA maintains stability.
Efficiency:
- Speedup: Achieved up to 2.5× prefilling speedup at 128K context compared to standard MLA.
- Memory Reduction: Reduced KV cache memory by 90% (e.g., from ~10GB to ~0.7GB at 128K).
- Scalability: The custom Triton kernel implementation allowed LCA to scale efficiently to 1M tokens, whereas PyTorch implementations of baselines ran out of memory (OOM) beyond 64K.

5. Significance and Impact

Bridging the Gap: LCA successfully bridges the gap between memory-efficient architectures (MLA) and computation-efficient methods (sparse attention), solving both bottlenecks simultaneously without reconstruction overhead.
Practical Deployment: By requiring no extra parameters and only lightweight fine-tuning, LCA is highly practical for deploying long-context models in resource-constrained environments.
Theoretical Rigor: The provision of a uniform error bound independent of context length provides a strong theoretical foundation for the reliability of latent-space condensation.
Future Direction: The work suggests that future long-context models should move beyond simple sparsification or compression and instead leverage the structural properties of latent representations for joint optimization.

In summary, LCA represents a significant step forward in making 100K+ token context modeling feasible and efficient, offering a robust balance between high performance, low latency, and minimal memory footprint.