💬 NLP

Learning Decomposed Contextual Token Representations from Pretrained and Collaborative Signals for Generative Recommendation

This paper proposes DECOR, a unified framework that addresses the objective misalignment between semantic tokenization and user interaction modeling in generative recommenders by learning decomposed contextual token representations that preserve pretrained semantics while adapting to collaborative signals, thereby outperforming state-of-the-art baselines.

Original authors: Yifan Liu, Yaokun Liu, Zelin Li, Zhenrui Yue, Gyuseok Lee, Ruichen Yao, Yang Zhang, Dong Wang

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifan Liu, Yaokun Liu, Zelin Li, Zhenrui Yue, Gyuseok Lee, Ruichen Yao, Yang Zhang, Dong Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart robot (a Large Language Model) how to recommend the perfect item to a customer, like a pair of headphones. To do this, the robot needs a way to "speak" about these items.

The Old Way: The Rigid Dictionary

Currently, most systems use a two-step process that is a bit like using a dictionary written by a librarian who has never shopped for headphones.

Step 1 (The Dictionary): First, a system looks at the item's description (e.g., "Noise-canceling headphones for sleep") and assigns it a fixed code, like a secret ID number. Let's say this code is [A_3, B_4, C_5, D_95]. This code is created once and never changes.
Step 2 (The Shopper): Then, the robot learns to predict the next item based on what the user has clicked before, using those fixed codes.

The Problem: This approach has two major flaws, which the paper calls "Suboptimal Static Tokenization" and "Discarded Pretrained Semantics."

The "One-Size-Fits-All" Trap: Imagine three different headphones: one for the office, one for the gym, and one for sleeping. They all sound similar, so the dictionary gives them the exact same first three codes: [A_3, B_4, C_5]. The robot sees these and thinks, "Oh, they are all the same!" It doesn't know if the user wants to sleep or run until it sees the very last digit. This confuses the robot and makes it slow to understand what the user actually wants.
The "Wasted Knowledge" Trap: The dictionary was built using a massive amount of general knowledge (like knowing that "Apple" can be a fruit or a tech brand). But once the robot starts learning from user clicks, it often forgets this general knowledge. It treats the code [A_3] as just a random symbol, throwing away the rich meaning it originally had.

The New Solution: DECOR

The authors propose a new framework called DECOR (Decomposed Contextual Token Representations). Think of DECOR as giving the robot a smart, adaptable translator instead of a rigid dictionary.

DECOR fixes the problems with two main tricks:

1. The "Contextual Translator" (Contextualized Token Composition)

Instead of treating the item code [A_3, B_4, C_5] as a fixed, unchangeable label, DECOR lets the robot reinterpret that code based on the current situation.

The Analogy: Imagine you hear the word "Bank." If you are talking about fishing, "Bank" means the side of a river. If you are talking about money, it means a financial institution.
How it works: In the old system, "Bank" always meant the same thing. In DECOR, if the user was just looking at fishing gear, the robot looks at the code for "Bank" and says, "Ah, in this context, this code means 'River Side'." If the user was looking at credit cards, it says, "In this context, this code means 'Money'."
The Result: The robot can instantly understand the intent behind the item, even if the item's code looks identical to other items. It stops being confused by the "prefix ambiguity" shown in the paper's Figure 1.

2. The "Dual-Engine Memory" (Decomposed Embedding Fusion)

DECOR realizes that the robot needs two types of memory working together, not just one.

Engine A (Frozen Knowledge): This is the original, rich knowledge from the dictionary (e.g., knowing that "Apple" is a fruit). DECOR keeps this part frozen so it never gets overwritten or forgotten.
Engine B (Learned Behavior): This is the new knowledge the robot learns from user clicks (e.g., "People who buy laptops often buy mice").
The Fusion: DECOR acts like a smart mixer. It takes the frozen knowledge (Engine A) and the new behavior patterns (Engine B) and blends them together perfectly. This way, the robot never loses its general smarts while still learning specific user habits.

Why It Matters

The paper tested this on three real-world datasets (Scientific papers, Musical Instruments, and Video Games).

Better Accuracy: DECOR consistently beat the best existing methods. For example, in the "Musical Instruments" category, where items have very specific technical names, DECOR improved accuracy by over 5% compared to the next best method.
Faster Learning: Because DECOR doesn't have to constantly rewrite the dictionary (which causes instability in other methods), it learns faster. It reached its peak performance 23 training sessions (epochs) faster than a competing method that tried to do everything at once.
Efficiency: It adds almost no delay to the recommendation process (less than 1 millisecond), making it fast enough for real-time use.

Summary

In simple terms, previous systems were like a librarian who gave every book a static, confusing label and then forgot the story inside the book. DECOR is like a librarian who keeps the original story intact but can instantly explain the label in a way that makes sense for the specific reader standing in front of them right now. It combines deep, pre-existing knowledge with real-time user behavior to make smarter, more accurate recommendations.

Technical Summary: Learning Decomposed Contextual Token Representations from Pretrained and Collaborative Signals for Generative Recommendation

1. Problem Statement

The paper addresses two critical limitations inherent in current two-stage generative recommendation paradigms, which typically involve item tokenization followed by Large Language Model (LLM) training:

Suboptimal Static Tokenization (C1): Existing methods rely on fixed semantic IDs generated by a pretrained tokenizer (e.g., RQ-VAE) to reconstruct item metadata. These static tokens fail to adapt to diverse user interaction contexts. Consequently, semantically distinct items (e.g., headphones for office vs. sleep) may share identical prefix tokens, creating "prefix ambiguity" that blinds the model to user intent until the final generation step. This static assignment creates a representation bottleneck that limits the discriminative power of the attention mechanism.
Discarded Pretrained Semantics (C2): During the recommender training stage, the rich semantic knowledge encoded in the pretrained embeddings is effectively discarded. While the tokenizer learns to reconstruct these embeddings, the token embeddings used by the LLM are randomly initialized and trained solely on user interaction data. This process overwrites the pretrained world knowledge (e.g., the dual meaning of "Apple" as a fruit or a tech brand), leading to misaligned recommendations, particularly for sparse items.

The authors note that existing attempts to solve these issues, such as end-to-end joint optimization (e.g., ETEGRec), often introduce training instability and computational overhead due to repeated re-tokenization.

2. Methodology: DECOR

The authors propose DECOR (Decomposed Contextual Token Representations), a unified framework designed to preserve pretrained semantics while enhancing token adaptability without requiring iterative re-tokenization. DECOR consists of two primary components:

3.1 Decomposed Embedding Fusion

To address the loss of pretrained semantics (C2), DECOR maintains two parallel embedding spaces:

Frozen Pretrained Semantic Embeddings ( $E_{pre}$ ): Derived from the codebooks of a pretrained RQ-VAE tokenizer. These are kept frozen to preserve the original semantic reconstruction capabilities.
Learnable Collaborative Embeddings ( $E_{collab}$ ): Trained from scratch based on user interaction sequences to capture sequential patterns and co-occurrence dynamics.

These two modalities are fused through a lightweight network. Both embeddings are projected into a shared latent space, normalized, concatenated, and mapped back to the original dimension via a learnable fusion matrix. This allows the model to integrate static semantic knowledge with dynamic collaborative signals adaptively.

3.2 Contextualized Token Composition

To address suboptimal static tokenization (C1), DECOR introduces a dynamic interpretation mechanism applied during the autoregressive generation phase. Instead of relying solely on the static embedding of a token, the model refines the token representation based on the user's interaction context:

Context Vector: An attention-based pooling mechanism aggregates the fused embeddings of the historical interaction sequence to generate a context vector ( $u_c$ ).
Soft Composition: For each token in the sequence, the model performs a soft composition over a candidate set of tokens (specifically, all tokens from the same RQ-VAE codebook layer). The final composed embedding is a weighted sum of candidate embeddings, guided by the context vector.
Residual Fusion: The contextually composed embedding is fused with the original static embedding via a residual link controlled by a hyperparameter $\alpha$ . This allows the model to reinterpret static tokens dynamically without altering the underlying tokenizer.

3.3 Learnable BOS (Beginning-of-Sequence) Composition

Recognizing that the first token acts as a global semantic anchor, the authors introduce a set of learnable BOS query vectors. These vectors are composed contextually to refine the initial prefix, ensuring that the generation process starts with a semantic anchor aligned with the user's specific history, thereby reducing ambiguity propagation.

3. Key Contributions

Problem Analysis: The paper is the first to explicitly analyze how static tokenization limits the ability of generative recommenders to adapt item representations to diverse user context dynamics, identifying prefix ambiguity as a key failure mode.
Novel Framework (DECOR): The authors propose a framework that fuses frozen pretrained semantic embeddings with learnable collaborative embeddings. Crucially, it introduces a dynamic, context-aware token composition mechanism that interprets static input tokens during generation, avoiding the instability of end-to-end re-tokenization.
Efficiency and Stability: The method achieves faster convergence and higher accuracy compared to joint training baselines, with only moderate computational overhead.

4. Experimental Results

The authors evaluated DECOR on three real-world datasets from the Amazon Review collection: Scientific, Instrument, and Game.

Performance: DECOR consistently outperformed state-of-the-art baselines, including traditional sequential recommenders (SASRec, GRU4Rec) and generative models (TIGER, LETTER, CoST, ETEGRec).
- On the Scientific dataset, DECOR achieved a relative improvement of over 14% in NDCG@10 compared to the strongest baseline.
- On the Instrument dataset (noted for specialized terminology and sparse text), DECOR improved NDCG@10 by 5.0% over the best baseline, demonstrating robustness where text-prior-based models struggled.
Ablation Studies: Removing either the Decomposed Embedding Fusion (DEF) or Contextualized Token Composition (CTC) resulted in significant performance drops, confirming that both preserving pretrained semantics and enabling contextual adaptation are necessary. The addition of Learnable BOS queries further enhanced performance, particularly in aligning high-level semantics with user context.
Convergence: DECOR reached 95% of its peak performance 23 epochs faster than the joint training baseline (ETEGRec) and avoided the learning stagnation observed in ETEGRec during intermediate training phases.
Embedding Utilization: DECOR significantly improved the utilization of codebook entries. While TIGER utilized only ~26% of active embeddings in the first quantization layer, DECOR activated 100% of embeddings on the Instrument and Game datasets by leveraging inactive tokens during composition.
Efficiency: Inference latency increased by approximately 0.85 ms per sample (at batch size 128) compared to the TIGER baseline, while maintaining ~82% of the baseline throughput. This trade-off is presented as highly favorable given the 5–14% accuracy gains.

5. Significance and Claims

The paper claims that DECOR provides a novel solution that avoids the instability of re-tokenization while retaining the benefits of pretrained semantics. By decoupling the static semantic structure from the dynamic collaborative signals and allowing for contextual reinterpretation at generation time, DECOR bridges the gap between world knowledge and user behavior. The authors position this as a more stable and expressive alternative to existing generative recommendation frameworks, particularly effective in scenarios with sparse data or specialized item terminologies where static text priors are insufficient.