Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation

1. Problem Statement

Large Language Models (LMs) are increasingly adapted to specialized domains (e.g., generative recommendation) by extending their vocabulary with thousands of new, learnable tokens (e.g., Semantic IDs or SIDs).

• The Bottleneck: The standard practice for initializing these new tokens is Mean-of-Vocabulary Initialization, where all new token embeddings are set to the mean of the existing vocabulary embeddings.
• The Failure Mode: The authors identify that this strategy causes a token-embedding collapse. By assigning identical vectors to all new tokens, the method erases inter-token distinctions and discards the semantic structure required for the tokens to function.
• The Limitation of Fine-Tuning: The paper demonstrates through spectral and geometric analysis that subsequent supervised fine-tuning (SFT) is insufficient to recover the lost semantic structure. The embeddings remain in a degenerate, low-rank subspace, preventing the model from effectively leveraging its pre-trained general knowledge for the new domain.

2. Methodology: Grounded Token Initialization (GTI)

To address the initialization bottleneck, the authors propose GTI, a lightweight, two-stage training pipeline.

Core Hypothesis

Grounded Token Initialization Hypothesis: Linguistically grounding novel tokens in the LM's pre-trained embedding space before fine-tuning enables the model to better leverage its general-purpose knowledge for the new domain.

The GTI Pipeline

1. Freeze the Backbone: The pre-trained LM backbone and original vocabulary embeddings are frozen. Only the embeddings of the newly introduced tokens ($E_{new}$) are trainable.
2. Paired Linguistic Supervision: A grounding stage is inserted prior to standard end-to-end fine-tuning. The model is trained on a paired dataset $(x_i, y_i)$ where:
   • $x_i$: Natural language description of an item (e.g., title, description).
   • $y_i$: The corresponding sequence of new Semantic ID tokens.
   • Bidirectional Training: The training includes both directions:
     • Text $\to$ Tokens: Given a description, generate the Semantic ID.
     • Tokens $\to$ Text: Given a Semantic ID, reconstruct the description.
3. Optimization: The model minimizes the negative log-likelihood over the new token embeddings ($E_{new}$) while keeping the rest of the network fixed. This forces the new tokens to map to distinct, semantically meaningful locations in the pre-trained manifold that align with the linguistic descriptions.
4. Downstream Fine-Tuning: After the grounding stage, the learned $E_{new}$ serves as the initialization for the standard supervised fine-tuning of the full model (or LoRA adapters) on the target task.

3. Key Contributions

1. Diagnosis of Misalignment: Through spectral analysis (Singular Value Decomposition) and geometric diagnostics (pairwise cosine similarity), the authors prove that mean initialization collapses new tokens into a degenerate subspace. They show that standard fine-tuning fails to recover the high-rank structure necessary for effective retrieval.
2. GTI Framework: They introduce GTI, a simple, efficient grounding stage that uses paired linguistic supervision to initialize new tokens. Unlike previous methods (like LC-Rec) that rely on multi-task learning during fine-tuning, GTI addresses the problem before fine-tuning begins.
3. Empirical Validation: The method is validated on Generative Recommendation (GR) benchmarks, a demanding task requiring the integration of thousands of new tokens. GTI consistently outperforms both mean initialization and existing multi-task adaptation baselines across industrial-scale and public datasets.

4. Experimental Results

The authors evaluated GTI on two datasets:

• Industrial Dataset: A candidate retrieval system from a major recruitment platform (Job-Candidate matching).
• Public Dataset: The Vibrent Clothes Rental dataset.

Key Findings:

• Performance Gains: GTI significantly outperforms the baseline (Mean Initialization + SFT) and the state-of-the-art LC-Rec (Mean Initialization + Multi-task SFT).
  • On the industrial dataset, GTI achieved a +21.63% relative gain in Precision@5 (Good Match) over the baseline, compared to only +6.38% for LC-Rec.
  • On the public dataset, GTI showed substantial improvements in Recall@K (e.g., +26.02% at Recall@20) compared to LC-Rec.
• Structural Preservation:
  • Spectral Analysis: GTI-initialized embeddings exhibit slower spectral decay and higher effective rank after fine-tuning, indicating a non-degenerate subspace.
  • Representational Similarity Analysis (RSA): GTI embeddings maintain a stronger correlation with the ground-truth RQ-VAE codebook structure compared to other methods.
  • Visualization: Cosine similarity matrices show that GTI preserves the blockwise hierarchical structure of Semantic IDs, whereas mean initialization results in a uniform, collapsed block even after fine-tuning.

5. Significance

• Redefining Vocabulary Extension: The paper shifts the focus from how to train new tokens during fine-tuning to how to initialize them. It establishes that initialization quality is a primary bottleneck in extending LMs.
• Efficiency: GTI is computationally efficient. It requires training only the new token embeddings (a small fraction of parameters) for a short duration before the main fine-tuning process, avoiding the complexity of multi-task objective alignment during the main training loop.
• Generalizability: While tested on Generative Recommendation, the principle of "linguistic grounding before fine-tuning" is applicable to any domain requiring vocabulary expansion (e.g., tool use, visual concepts, or specialized medical codes), offering a robust solution to the "cold start" problem for new tokens.

In conclusion, the paper argues that initialization is not just a starting point but a critical determinant of success in vocabulary extension. By grounding new tokens in the pre-trained linguistic manifold via paired supervision, GTI unlocks the model's ability to transfer general knowledge to novel domains effectively.