Technical Summary: Token Factory

Problem Statement

Large Recommendation Models (LRMs) based on Transformer architectures have shown significant promise in industrial-scale tasks, such as next-video prediction. However, a critical bottleneck exists in holistically integrating traditional signals (dense, sparse, and embedding features) into these models. Conventional approaches typically "textualize" these heterogeneous signals or map them to discrete item representations (e.g., Semantic IDs). This direct mapping leads to:

• Excessive Prompt Lengths: Converting dense signals (e.g., watch completion ratios, watch times) into text creates prohibitively long prompts.
• Resource Overhead: Long prompts result in substantial memory footprints and high computational costs during training and inference.
• Information Loss or Inefficiency: To fit within context windows, systems often truncate older user interactions, discarding valuable historical data.

Methodology: Token Factory

To address these limitations, the authors propose Token Factory, a framework that transforms traditional signals into "soft tokens." Unlike standard "hard" tokens (discrete IDs from a fixed vocabulary), soft tokens are continuous embedding representations generated directly within the model's embedding space.

Core Architecture

The framework consists of Token Makers, which act as specialized modules to process specific feature groups:

1. WH Token Maker: Processes user interaction sequences (e.g., watch history).
2. Query Token Maker: Integrates user-level features.
3. Candidate Token Maker: Integrates candidate item-level features.

Transformation Process:

• Input: Raw features (dense, sparse, embeddings) are normalized or embedded and concatenated into a vector $E_{input}$.
• Projection: A learnable function $G$ (e.g., a Multi-Layer Perceptron or neural network) maps $E_{input}$ to a fixed number ($N$) of soft tokens ($T_{output}$).
• Integration: These soft tokens are seamlessly interleaved with textual tokens (e.g., video titles) in the LRM prompt.
• Optimization: The function $G$ is co-trained end-to-end with the LRM, ensuring the soft tokens align with the model's semantic space.

Key Mechanisms

• Prompt Compactness: By mapping arbitrary numbers of features into a fixed, small number of soft tokens, the framework prevents prompt length explosion.
• Deterministic Length: The prompt length is capped at a predefined budget regardless of the input feature count, achieved via fixed-output projection layers.
• Sequence Compression: For long interaction histories, the framework employs MLP Compression (projecting the sequence dimension) or Attention Pooling (summarizing every $K$ items into one token) to further reduce sequence length while retaining critical information.
• Caching: Since user and query-level features are static across multiple candidate scoring, their corresponding soft tokens can be pre-computed and cached, significantly reducing inference latency.

Key Contributions

1. Empirical Validation of Traditional Signals: The authors demonstrate that leveraging a rich set of traditional signals (dense/sparse features) is critical for LRM performance, addressing limitations in current state-of-the-art methods that often exclude these due to computational trade-offs.
2. Novel Architectural Framework: Introduction of Token Factory, which enables the efficient mapping of heterogeneous features into soft tokens, serving as a new modality for recommendation models.
3. Efficiency and Quality Validation: Through offline and online evaluations, the framework is shown to achieve on-par or superior ranking and retrieval quality while enabling significant efficiency improvements.

Experimental Results

The framework was evaluated on PLUM, a production-scale Large Recommendation Model at Google, across ranking and generative retrieval tasks.

Ranking Task (CTR Prediction)

• Setup: Compared a baseline (textual Semantic IDs + custom tokenization of dense signals) against Token Factory (soft tokens).
• Efficiency: Token Factory reduced prompt length from 1,536 tokens to 480 tokens (a ~70% reduction), resulting in a 200% increase in training speed.
• Quality: Initially, the Token Factory model lagged slightly due to the need to learn new embedding tables. However, it caught up to the baseline at 1.5M training steps and maintained comparable performance.
• Scaling: Increasing the batch size by 200% (enabled by the efficiency gains) allowed the Token Factory model to outperform the baseline even in early training stages.

Generative Retrieval Task (Next Video Prediction)

• Setup: Compared a baseline (textual SIDs) against Token Factory on a YouTube Homepage recommendation task.
• Offline Performance: Achieved a +2.0% improvement in Recall@10.
• Online Performance:
  • Unique Impressions: Increased by +16.8%.
  • Fresh Content Discovery: Unique impressions for one-day-fresh videos increased by +67.1%, indicating improved ability to retrieve new content.
  • User Satisfaction: Positive gains in Satisfied Watchers (+0.04%) and Satisfied Watch Time (+0.05%), confirming that exploration gains did not compromise user experience.

Ablation Studies

• Feature Importance: Removing traditional dense/sparse features resulted in performance degradation, confirming their value.
• Soft Token vs. Textual SID: When context window constraints were tight, soft tokens outperformed textual SIDs because they allowed more items (200 vs. fewer) to fit within the fixed prompt budget.
• Longer History: Compressing extended watch histories (200 to 500 items) into soft tokens yielded a +0.08% AUC improvement, demonstrating the value of longer context retention.

Significance

The paper claims that Token Factory offers a unified solution to the "prompt length explosion" problem in industrial recommendation systems. By transforming heterogeneous signals into soft tokens, it allows Large Recommendation Models to:

1. Ingest richer information (longer histories, more features) without increasing computational overhead.
2. Maintain or improve model quality in both ranking and retrieval tasks.
3. Achieve significant efficiency gains in training and serving, enabling larger batch sizes and faster iteration.

The work positions soft tokens not merely as a compression technique, but as a necessary evolution for integrating the full spectrum of traditional recommendation signals into modern Transformer-based architectures.