🤖 machine learning

Tokens are All You Need: Dual-purpose Semantic IDs for Achieving LLM-Level I/O Efficiency in recommendation systems

This paper proposes Dual-purpose Semantic IDs, a method that uses hierarchical quantization to convert dense embeddings into discrete tokens for both collaborative identity and content reconstruction, thereby achieving LLM-level I/O efficiency and successfully deploying in production-scale recommendation systems to overcome memory bottlenecks.

Original authors: Baolei Li, Yiping Yuan, Yilin Zheng, Likang Yin, Ling Liu, Fabio Soldo, Romer Rosales, Xinyang Yi, Lichan Hong

Published 2026-07-29

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Baolei Li, Yiping Yuan, Yilin Zheng, Likang Yin, Ling Liu, Fabio Soldo, Romer Rosales, Xinyang Yi, Lichan Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the ultimate recommendation engine for a massive video platform, one that knows exactly what you want to watch next. To do this, the system needs to understand two things: who you are (your history) and what the videos are (their content). Traditionally, computers have stored this information as giant, heavy lists of numbers called "dense embeddings." Think of these like massive, high-resolution blueprints for every single video and user. While these blueprints are detailed, they are incredibly heavy to carry around. Every time the computer tries to make a suggestion, it has to haul these heavy blueprints from the storage room to the processor, creating a traffic jam known as the "Memory Wall." This slows everything down, especially when you have billions of users and videos.

On the other side of the tech world, Large Language Models (LLMs)—the brains behind smart chatbots—have figured out a trick. They don't use heavy blueprints; they use simple, discrete "tokens," like words in a sentence. These tokens are light, fast, and easy to process. The big question for recommendation systems has been: Can we swap those heavy, slow blueprints for these lightweight tokens without losing the rich details needed to make good suggestions? If we could, we might finally give recommendation systems the same speed and efficiency as the smartest AI chatbots.

This paper, titled "Tokens are All You Need," proposes a clever solution to this problem by introducing "Dual-purpose Semantic IDs." The authors, working at a major video sharing platform, suggest that we can compress those heavy video blueprints into a short sequence of tokens, much like turning a high-definition photo into a simple code. But here is the magic trick: these tokens do double duty. First, they act as a unique ID to help the system learn from user interactions (like which videos you clicked). Second, and more importantly, they can be instantly "decoded" back into a rough approximation of the original video details whenever the computer needs them.

The researchers tested this idea in a real-world production system with billions of examples. They found that by using this "on-the-fly" reconstruction method, they could drastically reduce the amount of data the system had to move around. In their experiments, this approach didn't just save space; it actually made the system faster and improved the quality of recommendations, particularly for new users or obscure videos that the system didn't have much data on yet. The paper suggests that by treating high-dimensional data as tokens, recommendation systems can break free from the "Memory Wall" and become as efficient as the most advanced AI models, proving that sometimes, you really do just need tokens to get the job done.

Technical Summary: Tokens are All You Need: Dual-purpose Semantic IDs for Achieving LLM-Level I/O Efficiency in Recommendation Systems

1. Problem Statement

Large-scale recommendation systems face a critical "Memory Wall" bottleneck caused by the reliance on massive, dense floating-point embedding tables. While Large Language Models (LLMs) scale efficiently due to their unified discrete token space and compute-bound nature, recommendation systems are constrained by the I/O and memory bandwidth required to ingest, store, and join high-dimensional continuous vectors (e.g., user history, content embeddings) during training and inference.

This limitation becomes particularly acute as systems evolve to handle sequential user activity with lengths scaling to $10^4$ or higher. Traditional approaches that attempt to incorporate rich content signals via dense embeddings suffer from prohibitive data footprints and serving latencies. Furthermore, while "Generative Retrieval" has introduced semantic tokens to replace categorical IDs, existing methods largely treat these tokens strictly as identifiers, failing to leverage them for efficient reconstruction of high-dimensional continuous content features.

2. Methodology: Dual-purpose Semantic IDs

The authors propose a framework that transforms high-dimensional continuous content embeddings into compact, discrete token sequences. This approach draws inspiration from computer vision data compression techniques (specifically VQ-VAE and VQGAN), proving that continuous spatial data can be compressed into discrete tokens without losing semantic meaning.

The core methodology consists of two concurrent roles for the generated Semantic IDs ( $S_i$ ):

A. Semantic ID Generation via Quantization

High-dimensional content embeddings ( $e_i \in \mathbb{R}^d$ ), typically derived from pre-trained multimodal models, are compressed into a sequence of $K$ discrete tokens using hierarchical quantization (e.g., Residual Quantization or RQ-VAE).
$S_i = [t_{i,1}, t_{i,2}, \dots, t_{i,K}]$
This reduces storage requirements from $d \times 32$ bits to $K \times \log_2(V)$ bits, achieving compression ratios of 50–100 $\times$ .

B. Dual-Purpose Framework

The framework utilizes these tokens for two simultaneous functions within the recommendation model:

Collaborative Identity (In-Graph Learning): The token sequence is treated as categorical features. The model learns embeddings for each token (or n-gram combinations) to capture user-item interaction patterns. Strategies include:
- Unigram: Independent token embeddings.
- Overlapping Bigram: Sliding window to capture local transitions.
- Nested N-gram: Hierarchical prefixes to enforce semantic clustering (e.g., all "Jazz" videos share a top-level embedding).
- Sentence Piece Model (SPM): Adaptive token combination based on data distribution.
  This component handles memorization and generalization, particularly for cold-start and long-tail items.
Content Reconstruction (SiDec): To recover the "pure" content signal without the I/O cost of joining dense vectors, the system employs a Semantic Decoder ( $f_\theta$ ).
- Process: The discrete tokens $S_i$ are looked up in a static codebook ( $\phi$ ) to retrieve latent embeddings, which are then passed through a lightweight decoder (MLP or shallow Transformer) to reconstruct an approximation of the original embedding ( $\hat{e}_i$ ).
- Integration: This reconstruction happens on-the-fly within the model graph. It replaces the need to store or log dense vectors in training data. The decoder can be frozen (using a pre-trained codebook) or trainable (to align with specific downstream tasks).

3. Key Contributions

Novel Dual-Purpose Framework: The paper introduces a system that addresses the "Memory Wall" by integrating standard Semantic ID learning with on-the-fly Semantic ID Decoding (SiDec). This balances item-specific memorization (via discrete tokens) with content-aware generalization (via reconstructed continuous semantics).
I/O Efficiency Breakthrough: By replacing massive vector storage with on-demand reconstruction, the framework drastically reduces data footprints and system overhead. It shifts the system burden from disk-bound dense vector retrieval to compute-bound on-the-fly reconstruction.
Production-Scale Validation: The authors provide extensive empirical evidence from a major video sharing platform (YouTube), demonstrating the framework's effectiveness in both ranking and retrieval models.

4. Experimental Results

The framework was evaluated through offline benchmarks and online A/B testing in production.

Offline Evaluation (Retrieval Model)

The study compared five experimental arms to analyze the trade-off between representation fidelity and training throughput:

Control: Standard IDs, no content embeddings (Highest throughput: 16.80 steps/s, lowest quality).
Arm 1 (Raw Dense): Direct ingestion of 64-dim embeddings (Quality improved, but throughput dropped 28.2% to 12.07 steps/s due to I/O bottlenecks).
Arm 2 & 3 (SiDec): Using codebook decoders (v0 and v1). These arms recovered throughput to ~15.3 steps/s (near Control levels) while maintaining or exceeding the quality of the raw dense approach.
Arm 4 (SiDec + Scaling): Combining the v1 codebook with architectural scaling achieved the best global loss (2.681) and Hit Rate @100 (0.2910), with a 20.4% throughput acceleration over the raw dense approach.

Conclusion: Discrete tokenization successfully breaks the I/O bottleneck, allowing for concurrent scaling of model depth and retrieval accuracy.

Online Deployment

The framework was deployed in multitask ranking and foundational transformer retrieval models.

Ranking Models: Adding the SiDec content reconstruction stream to existing Semantic ID features yielded significant gains in "Online Satisfied Engagement" (a composite metric of watch time and interactions).
- Watchpage Ranking: +0.80% improvement.
- Homepage Ranking: +0.22% improvement.
Retrieval Models: +0.13% improvement on the Homepage.
Impact: The improvements were statistically significant and disproportionately benefited nascent accounts with sparse histories and long-tail content, effectively alleviating popularity bias.

5. Significance and Claims

The paper claims that "Tokens are All You Need" for highly efficient, content-rich recommendations. The significance of this work lies in its philosophical and architectural shift:

Decoupling from Continuous I/O: The authors argue that high-dimensional continuous distributions do not need to be processed in their native floating-point format to retain predictive power. By quantizing the entire feature space (including user context, historical densities, and content embeddings) into a unified vocabulary of discrete tokens, recommendation systems can decouple from continuous floating-point I/O.
Alignment with LLM Scaling Laws: This approach aligns recommendation systems with the compute-bound hardware scaling laws enjoyed by LLMs, moving away from the memory-bound constraints of traditional dense embeddings.
Dual Utility: The framework demonstrates that discrete tokens can serve a dual purpose: acting as structured categorical features for collaborative filtering and as compressed representations for on-the-fly content reconstruction, eliminating the need for separate, heavy embedding tables.

The authors conclude that this paradigm offers a path to handle ultra-long user sequences and massive feature spaces without the prohibitive costs associated with traditional dense vector storage and retrieval.