💬 NLP

Do Value Vectors in Deep Layers Need Context from the Residual Stream?

The paper proposes "Bank of Values" (BoV), a novel attention mechanism for deep transformer layers that utilizes context-free, token-specific value vectors stored in a lookup table, demonstrating improved performance and efficiency over standard context-dependent value vectors across multiple model sizes and benchmarks.

Original authors: Muyu He, Yuchen Liu, Qingya Huang, Li Zhang

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muyu He, Yuchen Liu, Qingya Huang, Li Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant library where every book (a word or "token") has a specific set of instructions on how it should be understood in a story. In modern AI (Large Language Models), these books are read by a team of librarians (the "attention layers") who decide how much attention to pay to each book based on the context of the story so far.

Usually, these librarians take the book, look at the current story context, and then write a new set of notes (called a "value vector") for that book before passing it on. This process is like a librarian rewriting a book's summary every single time they read it, based on what happened in the previous pages.

The Big Discovery
The researchers in this paper found something surprising: for the librarians working in the back rooms of the library (the deeper layers of the AI), they don't actually need to rewrite the notes based on the story context.

Instead, these deep-layer librarians work best if they just grab a pre-written, perfect summary of the book that never changes, no matter what story is being told. It turns out that for these specific layers, the "original, context-free" information about the word is more valuable than the "context-dependent" notes they usually create.

The Solution: The "Bank of Values"
Based on this, the authors created a new system called Bank of Values (BoV).

The Old Way (Standard Attention): Every time the AI reads a word, it calculates a new, complex set of notes for that word, taking into account the whole story so far. This is computationally expensive and requires a lot of memory to store these temporary notes.
The New Way (BoV): For the deep layers, the AI stops calculating these notes on the fly. Instead, it has a giant, static "Bank" (a lookup table) where every possible word in the vocabulary has its perfect, pre-calculated note stored permanently.
- When the AI needs the note for a word, it simply looks it up in the bank.
- It's like having a dictionary where the definition is already written and printed, rather than having a team of writers sit down and compose a new definition every time you ask for a word.

Why is this better?

It's Faster and Cheaper: Because the AI doesn't have to do the heavy math of "rewriting" the notes for deep layers, it saves a massive amount of computing power (FLOPs).
It Saves Memory: In the old way, the AI had to keep a "cache" (a temporary holding area) of all the notes it had written so far for the current story. In the new way, since the notes are permanently stored in the "Bank," the AI doesn't need to carry around that heavy cache. It just grabs what it needs from the shelf.
It Works Better: The researchers tested this on AI models of different sizes. They found that using this "Bank" method actually made the AI smarter (lower error rates) and more efficient than the standard method, even though it was doing less work.

The Catch (and the Fix)
The paper notes that you can't do this for every librarian in the library. The librarians at the very front (the shallow layers) still need to look at the story context to understand the words. But for the librarians deep in the back, the "Bank" method is a game-changer.

In Summary
The paper argues that in the deep parts of an AI's brain, words don't need to be re-interpreted based on the story; they just need their original, pure identity. By storing these identities in a permanent "Bank" and looking them up instead of calculating them, the AI becomes faster, uses less memory, and performs better.

Technical Summary: Bank of Values (BoV)

Problem Statement
The transformer architecture, the backbone of modern Large Language Models (LLMs), relies on attention layers where query, key, and value vectors are computed from the residual stream. While the residual stream effectively accumulates context for queries and keys, the necessity of context for value vectors in deep layers remains under-examined. Standard attention computes value vectors ( $V = x_i W_V$ ) using the residual stream ( $x_i$ ), which mixes original token information with accumulated context. Recent works attempting to inject context-free token information into value vectors (e.g., Value Residual Learning, SVFormer) have shown mixed results, often failing to isolate the specific contribution of context-free components or applying them inefficiently across all layers. The core problem addressed is whether deep layers truly require context-dependent value vectors, or if they would benefit more from preserving original, context-free token information without the computational overhead of projecting the residual stream.

Methodology
The authors propose a systematic investigation into the composition of value vectors, followed by a novel architectural design called Bank of Values (BoV).

Systematic Ablation Study:
Using a 12-layer, 135M parameter model, the authors conducted experiments across four axes to determine the optimal composition of value vectors in deep layers:
- Additive vs. Substitutive: Comparing replacing the standard value vector entirely with a context-free source versus linearly combining them.
- Coefficients: Testing learnable (unbounded) coefficients against fixed ones.
- Source of Context-Free Information: Comparing a layer-specific projection of the initial embedding ( $x_0 W_V$ ) against a shared value vector from the first layer ( $V_1$ ).
- Target Layers: Evaluating the impact of applying these changes to all layers, every other layer, or only the last third of layers.
Key Findings from Ablation:
- Deep layers strongly prefer context-free components ( $\tilde{V}$ ) over context-dependent ones ( $V$ ). In additive variants, learned coefficients for $\tilde{V}$ were significantly larger (often >10) than those for $V$ (often driven to 0).
- Substituting $V$ with $\tilde{V}$ yields significant performance gains only in the last third of layers; applying this to shallow layers degrades performance.
- Once deep layers have access to a high-quality context-free component, adding the context-dependent component provides negligible benefit.
- Layer-specific context-free vectors ( $x_0 W_V$ ) outperform shared vectors ( $V_1$ ), as they allow each layer to project the token into a unique subspace.
Bank of Values (BoV) Architecture:
Based on the finding that deep layers benefit from context-free value vectors that are deterministic and independent of the input sequence, the authors propose BoV.
- Mechanism: In the last third of layers, the standard dense value matrix projection ( $x_i W_V$ ) is replaced by a sparse lookup table ( $E_V$ ).
- Implementation: For a token with ID $i$ , the value vector is retrieved directly as $v_i = \gamma_v E_V[i]$ , where $E_V \in \mathbb{R}^{|V| \times d}$ is a learnable parameter table storing pre-computed value vectors for every token in the vocabulary, and $\gamma_v$ is a learnable scaling coefficient.
- Inference Efficiency: Unlike standard attention, which caches value vectors for all previous tokens (requiring $O(Td)$ memory per layer), BoV does not cache values. Instead, it performs an $O(Td)$ gather operation from the static parameter table $E_V$ during the forward pass. The table $E_V$ can be offloaded to host memory, with only required entries prefetched.

Key Contributions

Insight: The paper demonstrates that value vectors in deep transformer layers primarily benefit from context-free, original token information, and that the context-dependent component derived from the residual stream is largely redundant in these layers.
Architecture: Introduction of Bank of Values (BoV), a method that replaces the computation of value vectors in deep layers with a static lookup from a token-specific table.
Efficiency: BoV eliminates the need to compute and cache value vectors for deep layers, reducing memory usage for long-context inference and lowering FLOPs per token.

Results
The authors evaluated BoV on models of 135M and 780M parameters, trained under compute-controlled budgets ( $1.50 \times 10^{18}$ and $3.91 \times 10^{19}$ FLOPs, respectively).

Validation Loss: BoV achieved lower validation loss compared to standard attention baselines on a held-out split of the ClimbMix dataset at both model scales.
Benchmark Performance: At the 780M scale, BoV improved the average score across 21 DCLM CORE benchmarks compared to the standard attention baseline.
Comparison: BoV matched or surpassed existing methods that inject token information into value vectors (e.g., Value Residual Learning) while using less memory and fewer FLOPs. Specifically, it outperformed methods that added token information on top of the residual stream computation, which still incurred the cost of computing and caching the context-dependent values.

Significance
The paper claims that BoV offers a more efficient alternative to standard attention for deep layers by leveraging the insight that these layers do not require context to compute value vectors. By persisting value vectors as sparse model parameters rather than dynamic computations, BoV reduces the memory footprint for long-context inference and computational cost without sacrificing (and in some cases improving) model performance. This challenges the default paradigm of computing all attention vectors from the residual stream and suggests a path toward more efficient transformer architectures.

Technical Summary: Bank of Values (BoV)

More like this