💬 NLP

Variable-Length Semantic IDs for Recommender Systems

This paper introduces a variable-length semantic ID framework for recommender systems using a discrete variational autoencoder with Gumbel-Softmax reparameterization to overcome the inefficiencies of fixed-length identifiers by adaptively assigning shorter descriptions to frequent items, thereby bridging the gap between generative recommendation and emergent communication principles.

Original authors: Kirill Khrylchenko

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kirill Khrylchenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive library with billions of books. Your job is to help a reader find the next book they'll love based on what they've read before.

In the old days, you would give every single book a unique, long, complicated barcode (like Book_98234712). To teach a computer to understand these books, you'd have to memorize every single one of these barcodes. If you have a billion books, that's a billion things to memorize! It's slow, clunky, and if a new book arrives that no one has seen before (a "cold" book), the computer has no idea what to do.

Recently, smart people started using Generative AI (like the technology behind chatbots) to solve this. Instead of memorizing barcodes, they teach the AI to "speak" about books using a special, simplified language. They call these Semantic IDs.

Here is the problem with the current way of doing this: Everyone gets the same amount of space.

The Problem: The "Fixed-Length" Suit

Imagine you are packing for a trip.

The Popular Item: You are packing a famous, famous celebrity's t-shirt. It's simple, everyone knows it, and it's easy to describe. You could just say, "The Star Shirt."
The Rare Item: You are packing a weird, obscure, one-of-a-kind antique spoon. It's hard to describe. You need a whole paragraph to explain its history, the material, and the weird handle shape.

Current AI systems force you to use the same suitcase size for both.

For the "Star Shirt," you have to fill a huge suitcase with empty space just to make it fit the rules.
For the "Antique Spoon," you have to cram it into that same suitcase, but you might not have enough room to describe all the details, so you lose information.

This is what the paper calls Fixed-Length Semantic IDs. It's inefficient. It wastes space on popular things and struggles with rare things.

The Solution: Variable-Length Semantic IDs

The author, Kirill, proposes a new way: Let the suitcase size change based on what you are packing.

This is inspired by how humans speak. Think about how we talk:

We say "Hello" (2 letters) to greet a friend.
We say "The weather is nice today" (5 words) to describe a day.
We say a long, complex sentence to explain a difficult concept.

We naturally use short words for common things and long descriptions for rare or complex things. This is a rule of language called Zipf's Law.

Kirill's system teaches the AI to do the same thing with items:

Popular Items (The "Star Shirts"): The AI learns to give these very short, punchy codes (e.g., just 2 or 3 tokens). This saves massive amounts of space.
Rare/Cold Items (The "Antique Spoons"): The AI gives these longer, more detailed codes (e.g., 5 or 6 tokens) so it can describe them accurately.

How Does It Work? (The Magic Trick)

To make this happen, the author built a special machine called a Discrete Variational Autoencoder (dVAE) with a "Gumbel-Softmax" twist.

The Encoder (The Packer): Looks at an item and decides, "Is this popular? Then I'll use a short code. Is this rare? Then I'll use a long code."
The Decoder (The Unpacker): Takes that short or long code and tries to rebuild the item's identity.
The Training: The AI is punished if it uses too many words (to keep it efficient) but rewarded if it describes the item perfectly. It finds the perfect balance: shortest possible description that still makes sense.

Think of it like a smart compression algorithm. It compresses the "popular" stuff heavily because it's easy to guess, and leaves the "rare" stuff uncompressed so it doesn't get lost.

Why Does This Matter?

The paper tested this on huge music and video datasets (like Spotify or TikTok). Here is what they found:

It's Faster and Cheaper: Because popular items get shorter codes, the AI can "read" more of a user's history in the same amount of time. It's like reading a book where the common words are written in shorthand.
It's Smarter: The AI handles new, rare items better because it isn't forced to squash them into a tiny box. It gives them the space they need to be understood.
It's More Stable: The author compared their method to an older, "gambling" style of training (called REINFORCE) often used in AI communication. The new method is like a steady, reliable train, while the old method was like a bumpy rollercoaster that often crashed.

The Big Picture

This paper bridges two worlds: Recommender Systems (how Netflix suggests movies) and Emergent Communication (how AI agents learn to talk to each other).

By realizing that AI should speak like humans—using short words for common things and long words for rare things—we can build recommendation systems that are:

More efficient (less computing power needed).
More accurate (better at finding what you like).
Better at handling new items (no more "cold start" problems).

In short: Stop forcing every item into the same-sized box. Let the AI pack efficiently, just like a human would.

1. Problem Statement

The paper addresses critical limitations in applying generative models to large-scale recommender systems, specifically regarding item representation:

Vocabulary Gap & Cardinality: Integrating Large Language Models (LLMs) with recommender systems is hindered by the massive cardinality of item catalogs (millions to billions). Directly using item IDs as tokens creates a vocabulary gap, as these IDs lack semantic alignment with natural language.
Inefficiency of Fixed-Length IDs: Existing solutions use Semantic IDs (sequences of low-cardinality tokens) to represent items. However, current methods assign a fixed length to every item. This is inefficient because:
- It ignores the skewed frequency distribution of real-world catalogs (Zipf's law), where popular items require less information to describe than rare, long-tail items.
- It wastes token budgets on popular items and limits expressiveness for cold-start items.
- It misaligns with natural language, where frequent concepts are described with shorter phrases (Zipf's Law of Abbreviation).
Training Instability: Previous attempts to learn variable-length discrete representations in related fields (Emergent Communication) often rely on REINFORCE (policy gradient) methods, which are known to be unstable and difficult to scale to large datasets.

2. Methodology

The authors propose a framework to learn Variable-Length (VarLen) Semantic IDs using a Discrete Variational Autoencoder (dVAE) with Gumbel-Softmax reparameterization.

Core Architecture

Generative Model: The model treats an item embedding $x$ $x$ as being generated from a discrete message $z$ $z$ (the semantic ID) and its length $L$ $L$ .
- $p(x, z, L) = p(L) p(z|L) p(x|z_{1:L})$
Shared Vocabulary: Unlike traditional hierarchical quantization (e.g., RQ-VAE) where each position has a distinct vocabulary, this method uses a single shared vocabulary across all positions, treating the ID as a natural language-like sequence.
Length Modeling:
- Prior: A truncated geometric distribution is used as a prior for message length $L$ , encouraging shorter messages.
- Stopping Mechanism: Instead of a special End-of-Sequence (EOS) token, the encoder predicts a stopping probability at each step. The length $L$ is determined dynamically based on these probabilities.
Encoder (Variational Inference):
- Uses Gumbel-Softmax to allow differentiable sampling from categorical distributions, enabling end-to-end training via backpropagation (avoiding the high variance of REINFORCE).
- Implements Residual Encoding with Soft Relaxation: Instead of hard vector quantization, the encoder computes a soft assignment over the vocabulary and subtracts the expected embedding from the residual, creating a differentiable analogue of residual quantization.
Decoder: A causal transformer reconstructs the item embedding from the generated semantic ID prefix.

Objective Function (ELBO)

The model is trained to maximize the Evidence Lower Bound (ELBO), decomposed into three terms:

Reconstruction Loss ( $L_{recon}$ ): Minimizes the error between the original item embedding and the reconstructed embedding based on the generated prefix.
Vocabulary Regularization ( $L_{vocab}$ ): Encourages the uniform usage of the shared vocabulary (preventing collapse to a few tokens) via KL divergence against a uniform prior.
Length Regularization ( $L_{length}$ ): A composite term containing:
- Length Penalty: Penalizes longer messages to encourage efficiency.
- Entropy Term: Prevents the model from collapsing to a single fixed length, promoting diversity in code lengths.

3. Key Contributions

Variable-Length Semantic IDs: The first systematic application of variable-length coding to recommender systems, adapting code length to item popularity (shorter codes for popular items, longer for cold/long-tail).
Stable Training Framework: Introduces a dVAE with Gumbel-Softmax for learning discrete, variable-length codes. This bridges the gap between the stability of variational methods and the discrete nature of semantic IDs, offering a more robust alternative to REINFORCE-based approaches.
Efficiency-Quality Trade-off: Demonstrates that variable-length IDs can significantly reduce the token budget required for popular items without sacrificing reconstruction quality, while providing more expressive codes for rare items.
Cross-Domain Synthesis: Successfully integrates concepts from Emergent Communication (Lewis signaling games, length penalties) into the architecture of modern Generative Recommender Systems.

4. Experimental Results

The method was evaluated on three large-scale datasets: Yambda (Music), VK-LSVD (Short Video), and Amazon Toys & Games.

RQ1: Efficiency of VarLen IDs:
- Reconstruction Quality: VarLen dVAE achieved reconstruction quality comparable to or better than fixed-length baselines (R-KMeans and fixed-length dVAE) while using significantly fewer tokens on average (e.g., reducing average length from 5 to ~2.3–3.6 depending on the dataset).
- Popularity Correlation: The model naturally learned to assign shorter codes to popular items (high interaction counts) and longer codes to cold items, aligning with Zipf's Law of Abbreviation without explicit supervision.
RQ2: Downstream Recommendation Performance:
- In sequential recommendation tasks (Next-Item Prediction), VarLen IDs outperformed fixed-length IDs in Recall@K and Coverage@K.
- Token Budget Efficiency: Under a fixed token budget (512 tokens), VarLen models could represent significantly more historical user events (up to ~50% more events) than fixed-length models, leading to better context modeling and higher recommendation accuracy.
RQ3: Comparison with REINFORCE:
- The dVAE approach proved significantly more stable than REINFORCE-based training. REINFORCE experiments suffered from reconstruction collapse and high variance, whereas the Gumbel-Softmax dVAE converged reliably.
RQ4: Scalability:
- The method scales well with increased vocabulary size and maximum sequence length. Larger vocabularies allowed for finer distinctions with fewer tokens, further improving efficiency.

5. Significance

This work provides a theoretically grounded and practical solution for the "vocabulary bottleneck" in generative recommender systems. By adopting variable-length semantic IDs:

Scalability: It enables the use of LLMs for recommendation without being constrained by the massive size of item catalogs.
Efficiency: It optimizes the token budget, allowing models to process longer user histories or more complex item descriptions within the same computational limits.
Cold-Start Handling: It provides a mechanism to allocate more representational capacity to rare and new items, improving their discoverability.
Future Direction: It establishes a new paradigm for "generative retrieval" where items are not just retrieved but generated as adaptive-length semantic sequences, bridging the gap between information retrieval and natural language processing.