💬 NLP

Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space

This paper introduces Compressed Convolutional Attention (CCA) and its variant CCGQA, novel attention mechanisms that perform operations in a shared latent space to simultaneously reduce parameters, KV-cache size, and FLOPs, thereby achieving significant speedups in training and prefill while maintaining or improving performance compared to existing methods like GQA and MLA.

Original authors: Tomas Figliolia, Nicholas Alonso, Rishi Iyer, Quentin Anthony, Beren Millidge

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tomas Figliolia, Nicholas Alonso, Rishi Iyer, Quentin Anthony, Beren Millidge

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to read a massive, 100,000-page novel to answer a single question.

In the world of AI, this is what a "Transformer" model does. It uses a mechanism called Attention to look back at every word it has read so far to understand the current word.

The problem? As the story gets longer, the AI's "memory" (called the KV-cache) grows huge, and the math it has to do to connect words together becomes exponentially harder. It's like trying to find a specific sentence in a library that keeps adding a new shelf for every single word you read. Eventually, the library is so big you can't fit it on your desk (GPU memory), and looking through it takes forever.

Existing solutions tried to fix this by:

Sharing notes: Making multiple heads of the AI share the same memory (GQA). This saves space but doesn't make the math faster.
Compressing notes: Writing a tiny summary of the notes instead of the full text (MLA). This saves space, but the AI still has to "unzip" the summary to read it, so the math speed doesn't improve much.

The New Solution: Compressed Convolutional Attention (CCA)

The authors of this paper propose a new method called Compressed Convolutional Attention (CCA). Here is how it works, using a simple analogy:

1. The "Tiny Notebook" Strategy

Instead of writing down every single detail of the story in a massive notebook, CCA forces the AI to write its notes in a tiny, compressed notebook from the very beginning.

Old way: Write the full sentence, then try to summarize it later.
CCA way: Only write the summary while you are reading. The AI does all its thinking, connecting, and math inside this tiny notebook.

Because the notebook is smaller (compressed), the AI has to do much less math to find connections. It's like trying to find a needle in a small shoebox instead of a giant warehouse. This makes the AI faster to train and faster to run, even on long stories.

2. The "Smoothing" Trick (Convolutions)

There was a fear that if you compress the notes too much, you lose important details (like the flavor of a story). To fix this, the authors added a special "smoothing" step.

Imagine you are summarizing a movie. If you just write "It was good," you lose the plot. But if you write a summary and then run it through a filter that highlights the most important emotions and connections (this is the "Convolution" part), you keep the essence without the bulk.

The paper adds a "mixing" step that blends the notes together before the AI makes its decision. This ensures that even though the notes are tiny, they are smart and rich with information.

3. The "Split Personality" (Value-Shift)

The AI also uses a trick called "Value-Shift." Imagine you are listening to a conversation. Usually, you hear what is being said right now. But this method also forces half the AI's attention to look at what was said a moment ago.

This gives the AI a sense of rhythm and flow, helping it understand how one sentence leads to the next, which makes the summaries even better.

Why is this a Big Deal?

The authors combined their "Tiny Notebook" (Compression) with the "Sharing Notes" idea (Grouped Query Attention) to create CCGQA.

Think of it like a super-efficient team:

Old AI: A team of 100 people, each carrying a giant backpack of notes. They are slow and heavy.
GQA/MLA: A team where some people share backpacks, or they carry smaller backpacks but have to stop and unpack them to read.
CCA/CCGQA: A team where everyone carries a tiny, high-tech tablet. They don't need to unpack anything. They can read, think, and write back instantly.

The Results

Speed: On powerful computers (H100 GPUs), this new method is 1.7 times faster at processing long texts (16,000 words) compared to the standard method.
Memory: It uses 8 times less memory to store the "notes" (KV-cache) without losing any intelligence.
Quality: Surprisingly, the AI actually gets smarter (lower error rates) than previous methods, even though it is doing less work.

The Bottom Line

This paper introduces a way to make AI models that can read entire books in seconds, without needing a supercomputer to hold the memory. It does this by forcing the AI to think in "highly compressed summaries" from the start, using clever math tricks to ensure those summaries are still perfect. It's the difference between carrying a library in your car versus carrying a single, magical index card that knows everything.

1. Problem Statement

Transformer models, particularly those handling long contexts (e.g., reasoning models), face two primary bottlenecks:

Quadratic Compute Complexity: Standard Multi-Head Attention (MHA) requires $O(S^2)$ operations where $S$ is the sequence length. This makes prefilling and training expensive.
Linear KV-Cache Growth: During autoregressive generation, the Key-Value (KV) cache grows linearly with sequence length and hidden dimension, creating massive memory-bandwidth bottlenecks that limit inference throughput.

Limitations of Existing Solutions:

Grouped Query Attention (GQA): Reduces KV-cache size by sharing Key/Value heads but does not reduce compute (FLOPs) for training or prefill.
Multi-Latent Attention (MLA): Compresses K/V into a shared latent space to reduce cache size. However, it requires up-projection matrices to reconstruct full dimensions for the attention operation, meaning it offers no compute savings over MHA/GQA and adds parameter overhead. It also complicates Rotary Position Embedding (RoPE) integration.
State-Space Models (SSMs): Eliminate quadratic complexity but often lack the expressivity of attention for complex reasoning tasks.

2. Methodology: Compressed Convolutional Attention (CCA)

The authors propose Compressed Convolutional Attention (CCA), a novel architecture that performs the entire attention operation within a compressed latent space.

Core Mechanism

Down-Projection: Queries ( $Q$ ), Keys ( $K$ ), and Values ( $V$ ) are projected from the residual dimension $E$ to a smaller latent dimension $\tilde{e} = E/C$ (where $C$ is the compression factor) using learnable matrices.
Latent Space Operations: Unlike MLA, CCA does not up-project back to dimension $E$ before attention. Instead, it performs attention entirely in the compressed space.
Key Innovations to Maintain Expressivity: To prevent performance degradation from compression, CCA introduces three specific operations on the compressed latents:
- Convolutional Mixing: Applies 1D convolutions across both the sequence and channel dimensions for $Q$ and $K$ . This enhances information transfer and smoothing, similar to causal convolutions in SSMs.
- Q-K-Mean: Adds the mean of pre- and post-convolution $Q$ and $K$ values back to the tensors. This acts as a skip connection and increases attention diagonal sparsity when combined with normalization.
- Value-Shift: Generates value vectors from both the current input and the previous token in the sequence, introducing an inductive bias that half the heads cannot "see" the present token (similar to RWKV's token-shift).
RoPE Integration: Because attention happens in the latent space, RoPE can be applied directly to the compressed keys and queries without needing special "RoPE heads" or separate caches, simplifying the architecture compared to MLA.

Extension: Compressed Convolutional Grouped Query Attention (CCGQA)

The authors combine CCA with GQA-style parameter sharing.

Mechanism: Within the compressed latent space, Key and Value heads are grouped and shared (e.g., 4 query heads share 1 compressed KV head).
Benefit: This allows for decoupled compression rates (e.g., compressing queries less than keys) and achieves an additional $2\times$ reduction in KV-cache size without performance penalties.

3. Key Contributions

Unified Compute and Memory Reduction: CCA is the first method to simultaneously reduce parameters, KV-cache size, and FLOPs (both training and prefill) by the compression factor $C$ .
Orthogonality of Strategies: The paper demonstrates that parameter compression (CCA/MLA) and parameter sharing (GQA) are orthogonal and can be combined (CCGQA) to create a smooth Pareto frontier for tuning models toward memory-bandwidth limits or compute limits.
Efficient Kernel Implementation: The authors implemented a fused CUDA kernel for CCA/CCGQA on H100 GPUs, fusing convolutions with Flash Attention-style online softmax.
RoPE Simplicity: CCA integrates RoPE seamlessly in the latent space, avoiding the complex RoPE cache management required by MLA.

4. Experimental Results

Experiments were conducted on dense (1B parameters) and Mixture-of-Experts (MoE) models trained on the Zyda2 dataset.

Performance vs. MHA/GQA/MLA:
- Dense Models: CCA outperforms MLA in loss while using significantly fewer FLOPs. CCGQA outperforms standard MHA despite having a much smaller KV-cache and fewer parameters.
- MoE Models: CCGQA achieves 8x KV-cache compression compared to GQA/MLA with no drop in performance compared to standard MHA. It outperforms all other attention methods at equal KV-cache compression rates.
Compute Efficiency:
- CCA reduces training and prefill FLOPs by the compression factor (e.g., $4\times$ or $8\times$ ).
- Latency: On H100 GPUs with a sequence length of 16k:
  - Prefill: CCA/CCGQA is 1.7x faster than MHA and 1.3x faster than GQA-8.
  - Backward Pass: CCA is 1.3x faster than MHA.
Ablation Studies:
- Sequential convolutions provide the largest performance gain.
- Q-K-Mean and Value-Shift provide smaller but consistent improvements in perplexity and evaluation scores.

5. Significance and Implications

Long-Context Feasibility: By reducing the quadratic compute cost by a factor of $C$ , CCA enables processing significantly longer sequences (e.g., $4\times$ longer for the same FLOP budget) without sacrificing model quality.
Training and Inference Speed: Unlike previous methods that only optimized for decoding (inference) speed, CCA accelerates training and prefill (the dominant cost in long-context workloads).
Scalability: The method is highly compatible with existing parallelism strategies (Tensor Parallelism and Context Parallelism) because it reduces the communication volume (latent width $E/C$ ) rather than just the cache size.
Future Direction: The paper suggests that performing additional non-linear operations (like convolutions) on compressed latents is a promising direction to recover expressivity lost during dimensionality reduction, potentially applicable to other sequence mixers beyond attention.

In summary, CCA represents a paradigm shift from "compressing the cache but keeping the compute" to "compressing the entire attention mechanism," offering a highly efficient solution for next-generation long-context transformers.