💬 NLP

MoVE: Mixture of Value Embeddings -- A New Axis for Scaling Parametric Memory in Autoregressive Models

This paper introduces MoVE (Mixture of Value Embeddings), a novel mechanism that decouples parametric memory from computational cost in autoregressive models by utilizing a global bank of learnable value embeddings with dynamic soft gating, thereby enabling scalable capacity improvements in both text and image generation without increasing active FLOPs.

Original authors: Yangyan Li

Published 2026-02-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yangyan Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Heavy Backpack" Dilemma

Imagine an AI model (like the ones that write stories or create images) as a student trying to learn a massive amount of knowledge.

In traditional AI models, if you want the student to know more facts (like the capital of every country or how to paint a realistic sunset), you have to make the student's brain physically bigger. You do this by adding more layers of neurons, which is like giving the student a heavier backpack.

The Catch: A heavier backpack makes the student slower. Every time they take a step (generate a word or a pixel), they have to carry the weight of that entire backpack. To get smarter, you have to pay a huge price in speed and energy.

The Solution: MoVE (The "Shared Library" System)

The authors introduce a new system called MoVE (Mixture of Value Embeddings). Instead of making the student's backpack heavier, they give the student a magic, shared library that sits right next to their desk.

Here is how it works:

The Global Library (The Value Embedding Bank):
Imagine a giant bookshelf containing millions of "concept cards." One card might say "how to draw a cat," another "the definition of 'justice'," and another "the texture of velvet." This library is shared by every part of the student's brain. Everyone can access the same cards.
The Smart Librarian (The Soft Gating Mechanism):
When the student needs to write a sentence or draw a picture, they don't carry the whole library. Instead, a tiny, fast "librarian" (a gating mechanism) looks at what the student is doing right now.
- If the student is writing about a cat, the librarian instantly pulls the "cat" card from the library.
- If they are writing about a sunset, the librarian pulls the "sunset" card.
- The librarian mixes these specific cards with the student's current thoughts.
The Result:
The student's brain (the main model) stays small and light. They don't need to memorize every single fact inside their own head. Instead, they just need to know how to find the facts in the shared library.
- Old Way: To know 1,000 facts, you need a huge brain.
- MoVE Way: You can know 1,000,000 facts by just adding more cards to the library, without making the brain any bigger or slower.

What the Paper Actually Tested

The researchers didn't just talk about this idea; they tested it in two main areas to prove it works:

Writing Text: They used the system to train models to write text. They found that models using MoVE made fewer mistakes and wrote better sentences than standard models of the same size. Even better, they could keep adding more "cards" to the library to make the model smarter, and it kept getting better without slowing down.
Creating Images: They tested it on models that generate images (like turning text descriptions into pictures). The MoVE models created clearer, more accurate images than the standard ones.

They also tested it on a special, high-speed version of AI (called MLA) that compresses data to save space. MoVE worked perfectly there too, proving it's a flexible tool that fits different types of AI "brains."

The Key Takeaway

The paper claims that MoVE breaks the old rule that "more knowledge = slower speed."

Think of it like this:

Standard AI: To learn more, you have to build a bigger, slower factory.
MoVE AI: You keep the factory the same size, but you build a massive, efficient warehouse next door. The factory workers can grab exactly what they need from the warehouse instantly.

This allows us to build "memory-dense" models—AI that is incredibly knowledgeable and rich in facts, but doesn't require a massive, expensive computer to run. The paper shows that by simply increasing the size of this "shared library," we can make AI smarter without the usual penalty of slowing it down.

Technical Summary: MoVE (Mixture of Value Embeddings)

1. Problem Statement

Autoregressive sequence modeling, the foundation of modern Generative AI, faces a fundamental structural bottleneck: the rigid coupling of parametric memory (the model's capacity to store factual knowledge or visual patterns) with computational cost. Traditionally, increasing a model's memory capacity requires deepening or widening the network, which incurs a proportional linear increase in active FLOPs during training and inference. This creates a tradeoff where "memory-dense" models—those with vast encyclopedic or visual knowledge—are prohibitively expensive to compute. The paper seeks to break this coupling to establish a new axis for scaling capacity that is orthogonal to network depth.

2. Methodology: MoVE

The authors introduce MoVE (Mixture of Value Embeddings), a mechanism that decouples memory from compute by augmenting the Attention mechanism's Value stream.

Core Architecture

Global Value Embedding Bank: MoVE introduces a learnable global tensor $E \in \mathbb{R}^{N_{vocab} \times M \times d}$ , representing a shared repository of "concept vectors" (e.g., linguistic definitions or visual attributes) accessible by all attention layers. Here, $M$ is the number of embedding slots per token.
Decoupling Roles: Unlike standard Transformers where Input Embeddings ( $W_I$ ) serve dual roles (generating attention patterns and carrying semantic content), MoVE separates these. $W_I$ remains compact for routing, while the Value Embedding Bank $E$ provides a high-capacity repository strictly for the Value stream.
Dynamic Mixing via Soft Gating: For each sequence step and attention head, the model computes a differentiable soft gating coefficient for every slot in the bank. The final value tensor $V_S^{(h)}$ is a weighted sum of the standard dense projection and the retrieved global embeddings:
$V_S^{(h)} = g_{t,0}^{(h)} \odot V^{(h)} + \sum_{i=1}^{M} g_{t,i}^{(h)} \odot M_{t,i}^{(h)}$
The gate $g_{t,0}$ controls the standard path, while $g_{t,i}$ controls the contribution of specific memory slots. This allows the model to dynamically balance local context with global parametric knowledge.
Shared Across Layers: Crucially, the embedding bank $E$ is shared across all $L$ attention layers. This creates "gradient highways," allowing concepts to be learned rapidly and robustly, effectively densifying the signal for parameters that would otherwise be sparse in a layer-wise configuration.

Integration with Efficient Architectures

The paper demonstrates that MoVE can be seamlessly integrated into Multi-Head Latent Attention (MLA). Instead of injecting memory into the full-rank Value vectors (which would negate MLA's efficiency), MoVE injects the global memory directly into the compressed low-rank latent space ( $c_{KV}$ ). This preserves the inference efficiency of MLA (via KV cache compression) while expanding the model's parametric memory capacity.

3. Key Contributions

New Scaling Axis: MoVE establishes a method to scale parametric memory independently of network depth. By increasing the number of embedding slots ( $M$ ), models can become "memory-dense" without increasing the computational depth or FLOPs.
Modality-Agnostic Design: The mechanism is validated across text and image generation, proving its applicability to diverse autoregressive modalities.
Synergy with MLA: The work shows that memory augmentation and inference compression (MLA) are complementary; MoVE can enhance the capacity of compressed latent spaces without materializing high-rank tensors.
Ablation Insights: The study isolates the contributions of the global shared bank versus gating mechanisms, finding that the global architecture is the primary driver of efficiency, while gating the standard path is essential for arbitrating between retrieved knowledge and local context.

4. Experimental Results

The authors conducted strictly controlled experiments using the nanochat framework for text and LlamaGen for images, comparing MoVE against Standard Transformers and Layer-wise Value Embeddings (LaVE).

Text Generation (FineWeb-Edu):
- MoVE consistently outperformed both Standard and LaVE baselines across model depths (D12, D20, D32).
- Scalability: While LaVE performance saturated quickly as memory was coupled to layer count, MoVE showed sustained improvements. For example, in the D12 model, increasing MoVE slots to $\times8$ (4L) reduced the validation Bits Per Byte (BPB) from 0.838 (Standard) to 0.797, a gain of 0.041.
- Efficiency: MoVE- $\times1$ (half the slots of LaVE- $\times2$ ) often matched or exceeded the performance of LaVE- $\times2$ , suggesting superior parameter utilization via global sharing.
Image Generation (ImageNet):
- In the GPT-B (111M) and GPT-L (343M) models, MoVE significantly improved image fidelity.
- For GPT-B, MoVE reduced the Fréchet Inception Distance (FID) from 6.53 (Standard) to 5.62.
- Notably, while the LaVE baseline degraded performance at the GPT-L scale (FID 3.77 vs. 3.47), MoVE maintained a clear lead with an FID of 3.10.
MLA Integration:
- Applied to Multi-Head Latent Attention, MoVE demonstrated monotonic scaling up to $\times32$ memory density.
- Due to MLA's compression, the $\times32$ MoVE setting added a parameter count comparable to a standard $\times1$ bank but achieved a substantial BPB gain (0.0136 on D12), confirming that memory scaling does not require deepening the network.

5. Significance and Claims

The paper claims that MoVE offers a distinct "scaling economy" where memory is cheap and intelligence (compute) remains fixed. By decoupling parametric memory from computational depth, MoVE enables the construction of memory-dense models that possess vast knowledge without the prohibitive cost of massive dense networks.

The authors position MoVE as a general-purpose booster for autoregressive modeling, allowing practitioners to scale capacity simply by increasing mixture components. They acknowledge limitations, noting that while MoVE reduces FLOPs, it introduces memory bandwidth costs, and that the current parameter efficiency per additional parameter is lower than standard dense scaling. However, the work establishes a new dimension for scaling that is orthogonal to network depth, offering a promising architectural primitive for future Generative AI systems.