Technical Summary: Tokenizing Numerical and Embedding Features for LLM RecSys

Problem Statement

While Large Language Models (LLMs) have emerged as powerful backbones for recommender systems due to their sequence modeling and semantic understanding capabilities, a fundamental mismatch exists between LLM input interfaces and industrial recommendation pipelines. LLMs primarily consume discrete textual tokens, whereas practical recommendation systems rely heavily on continuous numerical features (e.g., price, ratings) and dense embedding features generated by upstream engineering or pretrained encoders.

Existing strategies to bridge this gap—such as verbalizing signals into text, discretizing item representations, or appending external features only after LLM encoding—often obscure scale-sensitive numerical information, discard fine-grained continuous signals, or limit the interaction between textual and non-textual features within the model. This paper addresses the challenge of incorporating these heterogeneous, non-textual signals directly into the LLM embedding space without losing their continuous nature or requiring post-hoc processing.

Methodology

The authors propose a soft-token fusion framework that maps numerical and embedding features into "soft tokens" (continuous vectors) compatible with the LLM's standard token interface. This framework is instantiated within a shared-parameter LLM-based two-tower retrieval model.

1. Soft-Token Construction

The method converts non-textual features into soft token sequences using specific encoding and compression strategies:

• Embedding Features: Dense embedding features (derived from item metadata or user history) are first encoded by an auxiliary LLM text encoder (Qwen3-4B). To manage sequence length, mean pooling is applied over consecutive token embeddings. These pooled embeddings are then compressed into a fixed number of embedding soft tokens using a Q-Former adapter. The Q-Former utilizes learnable query tokens to attend to the variable-length input features, acting as an information bottleneck.
• Numerical Features: Continuous numerical features (specifically price) are encoded using Fourier feature encoding to preserve fine-grained value differences and scale sensitivity. These encoded values are projected into the LLM hidden dimension and subsequently compressed into numerical soft tokens via a dedicated numerical Q-Former.

2. Interaction-Based Fusion

A critical component of the framework is the interaction-based fusion module. Rather than simply concatenating textual tokens, embedding soft tokens, and numerical soft tokens (which the authors found to be suboptimal), the proposed method refines the heterogeneous soft tokens before insertion.

• The embedding and numerical soft tokens are concatenated and passed through a lightweight Transformer fusion module.
• This module models relations between the two feature types using attention mechanisms (bidirectional or one-way cross-feature attention).
• The refined soft tokens are then inserted into the final LLM input sequence (Qwen3-0.6B) alongside the textual prompt tokens and the end-of-sequence ([EOS]) token.

3. Architecture

The system employs a two-tower retrieval architecture where both the user and item towers share the same LLM parameters. The towers encode the fused input sequences into dense retrieval representations using last-token pooling over the [EOS] token. The model is trained with a contrastive objective using in-batch negatives.

Key Contributions

1. Identification of Mismatch: The paper explicitly identifies the disconnect between token-based LLM inputs and the continuous numerical/dense embedding features standard in recommendation pipelines.
2. Soft-Token Fusion Framework: It proposes a method to convert non-textual features into LLM-compatible soft tokens, allowing heterogeneous signals to be consumed through the standard token interface.
3. Interaction-Based Fusion: The authors introduce a specific module that models interactions between embedding-derived and numerical soft tokens prior to LLM encoding, demonstrating that this is superior to direct concatenation.
4. Empirical Validation: The framework is instantiated in a shared-parameter two-tower retrieval model and evaluated on three Amazon recommendation benchmarks (Beauty, Sports and Outdoors, Toys and Games).

Experimental Results

The authors evaluated their method against several baselines, including text-only LLM variants, single-source soft-token variants, direct concatenation baselines, and established sequential/generative recommendation models (e.g., SASRec, TIGER).

• Effectiveness of Non-Textual Features: Both numerical and embedding soft tokens provided performance gains over text-only baselines. The relative utility varied by dataset (numerical features were more impactful on Beauty and Sports; embedding features were more impactful on Toys).
• Superiority of Interaction: Direct concatenation of heterogeneous soft tokens was found to be inconsistent and often underperformed single-source variants. In contrast, the interaction-based fusion approach achieved the best or tied-best performance across most metrics (Recall@K, NDCG@K) on all three datasets.
• Comparison with Baselines: The proposed interaction-based model outperformed the TIGER baseline in Recall@10 across all datasets and achieved the strongest overall performance on the Toys and Games dataset. It also surpassed traditional sequential models like GRU4Rec, Caser, and HGN.
• Ablation Studies: Additional experiments confirmed the necessity of specific design choices:
  • Replacing Q-Former adapters with MLP projections significantly degraded performance (up to 60% drop in R@5 on Sports).
  • Replacing Fourier encoding with tokenizer-based numerical representation consistently reduced performance.
  • Naive prompt concatenation was less effective than explicit soft-token construction and interaction.

Significance and Claims

The paper claims that soft-token fusion serves as a practical interface for bridging LLMs with feature-rich recommender system pipelines. By representing continuous numerical and dense embedding features as soft tokens and explicitly modeling their interactions before LLM encoding, the method allows LLM-based recommenders to exploit fine-grained non-textual signals without abandoning the token-based architecture.

The authors position this work as a step toward moving from task-specific recommendation architectures toward LLM-centered models that can natively consume heterogeneous data types. They note that future work could extend this framework to additional feature types, larger backbone models, and more realistic large-scale retrieval settings. The results suggest that modeling relations among non-textual features is crucial for maximizing the potential of LLMs in recommendation tasks.