🤖 machine learning

Do Transformers Need Three Projections? Systematic Study of QKV Variants

This paper systematically demonstrates that sharing query, key, and value projections in Transformers—particularly the Q-K=V variant—significantly reduces inference memory and KV cache requirements while maintaining competitive performance across vision and language tasks, thereby enabling efficient on-device deployment.

Original authors: Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Transformer model (the brain behind modern AI) as a highly efficient librarian trying to answer a question based on a massive book. To do this, the librarian uses three specific tools for every word in the book:

The Query (Q): A question card asking, "What am I looking for?"
The Key (K): A label on the book's spine to see if it matches the question.
The Value (V): The actual page content to read if the match is good.

Traditionally, the librarian creates a brand new, unique tool for every single word for each of these three steps. This is the standard "QKV" setup. It works great, but it's heavy. The librarian has to carry three heavy toolkits for every word, which takes up a lot of space in their backpack (memory) and slows them down.

This paper asks a simple, bold question: "Do we really need three separate tools, or can we combine them?"

The researchers tested three ways to simplify the librarian's toolkit:

The Three Experiments

1. The "Same Question, Same Label" Approach (Q = K - V)

The Idea: The librarian uses the same tool for asking the question and checking the label. They still have a separate tool for reading the page.
The Result: This is the big winner. It turns out that the "Question" and the "Label" tools are so similar that they can be the same object without hurting performance.
The Benefit: The librarian now only needs to carry two tools instead of three. This cuts the memory needed to store the "labels" and "pages" in half. It's like realizing you can use your left hand to both hold the map and point at the destination, saving you from carrying a second map.

2. The "Same Question, Same Page" Approach (Q - K = V)

The Idea: The librarian uses the same tool for the label and the page content, but keeps a separate tool for the question.
The Result: This also works well, but it's slightly less efficient for language tasks than the first option. It's like using the same key to unlock the door and open the safe, which is clever, but the "Question" tool still needs to be unique to keep the direction of the search correct.

3. The "One Tool for Everything" Approach (Q = K = V)

The Idea: The librarian tries to use just one single tool to ask the question, check the label, and read the page.
The Result: This is a disaster for language tasks. It's like trying to use a hammer to ask a question, check a label, and read a book all at once. The AI gets confused because it loses the ability to tell "what I'm looking for" apart from "what I found." The performance drops significantly.

Why This Matters: The "Backpack" Problem

The most exciting part of this paper isn't just about making the AI smarter; it's about making it lighter.

When an AI generates text (like writing a story or answering a chat), it has to remember everything it has written so far. This memory is called the KV Cache.

Standard AI: Carries a heavy backpack with separate compartments for "Labels" and "Pages."
New AI (Q-K=V): Carries a backpack where the "Labels" and "Pages" are merged into one compartment.

The Real-World Impact:

Double the Memory: Because the backpack is lighter, you can fit twice as many words in the AI's memory without running out of space.
Cheaper Servers: If you are running a company that uses AI, you can serve twice as many customers on the same number of computers. The paper calculates this could save companies thousands of dollars a month.
Running on Phones: This efficiency makes it much more realistic to run powerful AI models directly on your phone or a small edge device, rather than needing a giant supercomputer in a data center.

The "Double Dip" Bonus

The paper also discovered that this new "lighter backpack" trick works perfectly alongside another existing trick called Head Sharing (used by models like Llama 2 and Mistral).

Head Sharing is like having fewer librarians but making them work harder.
Projection Sharing (this paper's idea) is like making each librarian's toolkit smaller.

When you combine them, you get a massive win. The researchers showed that by combining these two methods, you can shrink the memory needed by 97%. This is the "holy grail" for running AI on small devices.

Summary

The paper proves that the standard AI design is slightly over-engineered. By merging the "Question" and "Label" tools into one, we can cut the memory cost in half with almost no loss in intelligence. It's a simple tweak that makes AI faster, cheaper, and ready to run on your pocket device.

Technical Summary: Do Transformers Need Three Projections?

Problem Statement

Since the introduction of the Transformer architecture, the Query, Key, and Value (QKV) attention mechanism has been the standard for aggregating information across sequences. However, the necessity of maintaining three distinct, independent linear projections for these components remains an open question. While other architectures like CNNs and State Space Models (SSMs) often utilize more unified internal representations, Transformers persist with a tripartite structure. As context windows expand and the demand for real-time inference grows, the research community has focused on architectural efficiency to alleviate the quadratic bottleneck of self-attention. This paper investigates whether the three projections are truly necessary or if they represent a redundancy that can be reduced to lower parameter counts and, more critically, inference memory overhead without significant performance degradation.

Methodology

The authors propose and systematically evaluate three "Projection-Sharing" constraints that progressively reduce the number of learned transformations in the attention mechanism:

Q=K-V (Unified Query-Key): The query and key projections are unified ( $Q=K$ ), while the value projection remains separate. This results in a symmetric attention matrix ( $KK^T$ ). To address the limitations of symmetry in sequential tasks, the authors introduce a variant $(Q=K-V)^+$ that injects asymmetry via 2D positional encodings.
Q-K=V (Unified Key-Value): The key and value projections are unified ( $K=V$ ), while the query remains separate. This preserves asymmetric attention maps ( $QK^T$ ) and imposes a form of weight tying between keys and values.
Q=K=V (Single Projection): All three roles share a single projection matrix. This combines the symmetry of the first variant with the representational bottleneck of the second.

The study also explores the orthogonality of these projection-sharing strategies with existing "Head-Sharing" techniques like Grouped Query Attention (GQA) and Multi-Query Attention (MQA). The authors hypothesize that combining projection sharing (constraining $K=V$ ) with head sharing (reducing the number of KV heads) can yield compound gains in memory efficiency.

Experiments were conducted across 12 diverse tasks:

Synthetic Reasoning: Five tasks involving list manipulation (Reverse, Sort, Sub, Swap, Copy).
Computer Vision: Image classification on MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and TinyImageNet, as well as set anomaly detection and semantic segmentation.
Language Modeling: Pre-training of 300M and 1.2B parameter GPT-style models on 10 billion tokens from the SlimPajama dataset.

Key Results

1. Performance vs. Efficiency Trade-offs

Q-K=V (Optimal Balance): This variant emerged as the clear winner. In language modeling, it achieved a 50% reduction in KV cache size with only a 3.1% degradation in perplexity (for 300M models) and 2.48% (for 1.2B models). It maintained competitive performance on vision and synthetic tasks.
Q=K-V (Symmetry Limitation): While this variant achieved training quality comparable to the baseline, it offered zero inference memory benefits because it still requires caching separate K and V tensors. Furthermore, the symmetric attention maps limited its effectiveness in causal language modeling, resulting in a 4.9% perplexity degradation.
Q=K=V (Over-constrained): This aggressive simplification caused catastrophic performance loss (25.4% perplexity degradation in language modeling), indicating that forcing a single projection for all three roles is too restrictive for complex tasks.

2. Synergy with Head Sharing

The study demonstrates that projection sharing and head sharing are strictly complementary.

Combining Q-K=V with GQA-4 (4 groups) resulted in an 87.5% cache reduction with a 3.9% perplexity degradation.
Combining Q-K=V with MQA (1 group) achieved a 96.9% cache reduction with a 4.8% perplexity degradation.
At the 1.2B scale, the Q-MQA configuration reduced the KV cache by 67× (from 5.9 GB to 88 MB per user for 32k context) while maintaining near-parity with the baseline in downstream task accuracy.

3. Scaling Behavior

The relative rankings of the variants remained stable as model size increased from 300M to 1.2B parameters. Notably, larger models appeared slightly more robust to projection constraints, with the perplexity degradation for Q-K=V dropping from 3.1% at 300M to 2.48% at 1.2B.

Significance and Claims

The paper claims that projection sharing is an underexplored instance of weight tying that offers direct, quantifiable inference memory benefits, which are particularly valuable for edge deployment and long-context applications.

Why Q-K=V Works: The authors argue that keys and values can occupy similar representational spaces, and attention operates effectively in a low-rank regime where the distinction between K and V is less critical than the asymmetry between Q and the shared K-V representation. Empirical analysis showed high cosine similarity (0.73) between standard K and V projection matrices, suggesting inherent redundancy.
Why Q=K-V Fails: Forcing $Q=K$ creates symmetric attention maps, which breaks the directionality required for causal dependencies in language modeling.
Practical Impact: The primary contribution is not just parameter reduction (which is modest, ~7-14% total model size reduction) but the 50% reduction in KV cache memory. This enables:
- 2× longer context windows for the same memory budget.
- 2× higher throughput (concurrent users per GPU).
- Significant cost reductions (e.g., a projected 43% reduction in serving costs for a code completion model).

The authors conclude that while no single architecture wins across all domains, the Q-K=V configuration provides the best quality-efficiency trade-off for sequential tasks, and its combination with head-sharing techniques enables practical on-device inference for billion-parameter models. The code is publicly available to facilitate further research.