Technical Summary: TokenMinds

Problem Statement

Industrial recommender systems traditionally rely on dense, fixed-dimensional user embeddings to model user interests. While effective, these representations suffer from inherent constraints, such as the loss of fine-grained signals when compressing a user's full spectrum of interests into a single vector. Conversely, emerging approaches using Large Language Models (LLMs) to generate text-based user profiles face significant bottlenecks: they often capture only topical co-occurrences rather than deep sequential behavior dynamics, struggle to ground text outputs to specific item attributes, and introduce modality gaps when integrated into non-textual downstream systems. Furthermore, while Semantic IDs (SIDs) have proven effective for item tokenization in generative recommendation, their application to user modeling remains largely unexplored.

Methodology

TokenMinds is an industrial-scale system designed to bridge these gaps by extending the PLUM framework from item retrieval to user modeling. It employs a unified encoder-decoder architecture adapted from pre-trained LLMs to generate two complementary outputs simultaneously:

1. Dense User Embeddings: Extracted from the encoder, ensuring compatibility with existing downstream models that rely on continuous vectors.
2. Discrete SID-Based User Tokens: Generated autoregressively by the decoder, providing semantically grounded, discrete representations of user interests.

Core Components

• Architecture: The system uses an encoder-decoder setup. The encoder processes sequential user features (watch histories, search queries) to produce dense embeddings. The decoder attends to the encoder's output via cross-attention to generate SID-based user tokens. This dual-output design allows the system to serve both legacy dense-embedding models and new token-based models.
• Input Tokenization: User behaviors are tokenized into a sequence comprising:
  • SID Tokens: Hierarchical discrete codewords derived from video content features via a Residual-Quantized Variational AutoEncoder (RQ-VAE). To encourage diversity and mitigate memorization, only the prefix-$L$ of the full SID is used.
  • Soft Tokens: Non-SID features (e.g., watch time, device) are encoded as soft tokens via MLP projection.
  • Condition Tokens: Scenario-specific tokens (e.g., <LFV>, <SFV>, <Search>) are prepended to distinguish content types and interleaved search queries.
• Training Objective: The model is trained via Supervised Fine-Tuning (SFT) following Continued Pre-Training (CPT) on the SID vocabulary. It employs look-ahead sampling, randomly selecting multiple future target watches from a 24-hour window rather than just the immediate next item. This prevents overfitting to immediate sessions and improves generalization to future interests. The loss function minimizes the cross-entropy of predicted SID prefixes, weighted by an engagement reward signal.
• Cross-Scenario Modeling: TokenMinds unifies Long-Form Video (LFV) and Short-Form Video (SFV) modeling within a single framework. By prepending scenario-specific condition tokens, a single model can process chronologically interleaved sequences. At inference, multi-context decoding allows the system to generate scenario-specific user tokens from a single shared encoder pass, eliminating the need for redundant inference runs.
• Serving Infrastructure: To handle billions of users under strict latency constraints, the system decouples representation generation from real-time scoring. User embeddings and tokens are generated asynchronously via a background service and cached in a key-value store. Real-time scoring retrieves these pre-computed representations, ensuring constant latency regardless of the underlying model complexity.

Key Contributions

1. Dual-Output Architecture: The proposal of TokenMinds, a unified architecture that jointly generates standard dense continuous embeddings and discrete SID-based user tokens, enabling compatibility with existing systems while introducing new representational capabilities.
2. Tokenized User Representations: The novel application of SID-based discrete tokens for user representation. The paper identifies optimal adaptation strategies (specifically Learnable Embeddings) for integrating these tokens into downstream ranking models, demonstrating that tokens and dense embeddings capture complementary signals.
3. Cross-Scenario Modeling: A highly efficient paradigm that consolidates training and serving for distinct scenarios (LFV and SFV) into a single model. This approach utilizes multi-context decoding to achieve scenario-aware outputs without multi-model overhead.
4. Industrial-Scale Deployment: The successful validation of the system through extensive offline experiments and live A/B tests on multiple YouTube surfaces, serving full user traffic (billions of users).

Results

• Offline Performance: Ablation studies confirm that look-ahead sampling and coarse-grained SID prefixes are critical for generalization. The use of CPT initialization significantly outperforms random initialization or standard pre-trained checkpoints.
• Online Performance: Live A/B tests on production ranking systems show statistically significant gains:
  • Core User Metrics: Up to +0.11% improvement.
  • Core Engagement Metrics: Up to +0.62% improvement.
  • Complementarity: Combining both dense embeddings and discrete tokens yields amplified gains compared to using either alone, validating the dual-output design.
• Efficiency:
  • Training: Consolidating LFV and SFV into a single model reduces upstream training compute by 50%.
  • Serving: Multi-context decoding reduces upstream serving compute by 31% compared to maintaining separate models.
  • Storage: Discrete token representations require 72% less storage (1,280 bytes vs. 4,608 bytes) than dense embeddings.
• Scalability: Scaling studies indicate that model performance saturates at approximately 1,000 watch history items, and larger batch sizes accelerate convergence and improve representation quality.

Significance

The paper claims that TokenMinds demonstrates the practical viability of SID-based user tokens at an industrial scale. It establishes that discrete, semantically grounded user representations offer a promising direction for scaling user modeling beyond the constraints of dense embeddings alone. By successfully integrating these tokens into production ranking systems and showing their complementary value with dense embeddings, the work suggests a new paradigm for user understanding in large recommender systems that balances generalization, efficiency, and compatibility.