Technical Summary: REPREC

Problem Definition

Sequential recommendation aims to predict a user's next interaction based on the chronological order of their past behaviors. While classical models (e.g., collaborative filtering, SASRec, BERT4Rec) effectively learn compact user representations from interaction sequences, they lack the semantic world knowledge and instruction-following capabilities of Large Language Models (LLMs). Conversely, recent approaches that reformulate recommendation as a natural language task using LLMs often struggle to preserve structured collaborative signals.

Existing solutions attempting to bridge this gap typically rely on one or more of the following strategies, which introduce significant deployment trade-offs:

• LLM Fine-tuning: Adapting the LLM backbone (e.g., via LoRA) to learn recommendation tasks.
• Architectural Modifications: Adding dedicated cross-attention modules or joint optimization of multiple components.
• Item-Level Conditioning: Injecting a representation for every historical item into the LLM input, causing sequence lengths and computational costs (FLOPs) to scale linearly with interaction history.
• Representation Distillation: Transferring knowledge via complex distillation objectives.

These methods often increase training complexity, parameter footprints, and inference latency, making them less suitable for production environments where modularity, scalability, and low latency are critical. The paper posits a design question: Can a frozen sequential encoder and a frozen LLM be connected solely through lightweight, user-level representation alignment, preserving modularity and minimizing deployment complexity?

Methodology

The authors propose REPREC (Representation-driven Parameter-Efficient Recommendation), a framework that connects a frozen sequential recommender and a frozen LLM via a lightweight, trainable alignment module. The architecture consists of three primary components:

1. Frozen Sequential Encoder ($f_\theta$): A standard sequential recommender (instantiated as SASRec or BERT4Rec) trained on next-item prediction. Once trained, its parameters are frozen. It maps a user's interaction sequence $S_u$ into a fixed-size $d$-dimensional user embedding.
2. Lightweight Injector ($g_\phi$): A compact Multi-Layer Perceptron (MLP) that projects the fixed-size user embedding into the LLM's embedding space.
   • The injector maps the $d$-dimensional embedding to a set of $m$ "soft tokens" (conditioning vectors) of dimension $D$ (the LLM hidden size).
   • The transformation involves a linear projection, a non-linear activation (GELU), and a shared LayerNorm applied to the resulting tokens.
   • Crucially, only the injector parameters $\phi$ are trainable; the sequential encoder and LLM remain frozen.
3. Frozen LLM Backbone ($M$): A pre-trained LLM (e.g., LLaMA, Qwen) that receives the injected soft tokens prepended to a textual prompt describing the candidate item. The LLM performs a binary classification (Yes/No) to determine if the candidate item is relevant.

Training Process:
The system is trained using a binary decision prompt. The user tokens $Z = g_\phi(u)$ are concatenated with the prompt embedding $E_{prompt}$ to form the input $E_{input} = [Z; E_{prompt}]$. The LLM generates logits for the answer token, and cross-entropy loss is computed only at the answer position. Gradients flow back through the frozen LLM activations to update only the injector parameters $\phi$.

Key Contributions

• Parameter Efficiency: REPREC eliminates the need to fine-tune the LLM backbone or the sequential encoder. Adaptation is confined to a small MLP injector, significantly reducing the trainable parameter count compared to LoRA or full fine-tuning.
• Modularity and Compatibility: The framework is encoder-agnostic and backbone-agnostic. It can integrate with various sequential encoders (SASRec, BERT4Rec) and LLMs (LLaMA, Qwen) without modifying their internal architectures.
• User-Level Alignment: Unlike item-level conditioning methods that scale with history length, REPREC uses a fixed number of soft tokens ($m$) derived from a global user representation, ensuring constant inference latency regardless of interaction history size.
• Generalization to Sparse Data: The method is specifically designed to transfer collaborative signals from the sequential encoder to the LLM, which is particularly beneficial for users with limited interaction histories.

Experimental Results

The authors evaluated REPREC on five Amazon product review datasets (Beauty, Sports, Toys, Pet Supplies, Tools) against baselines including Zero-shot LLM, Frozen Injector, LoRA-FT, and LLaRA.

• Overall Performance: REPREC consistently outperforms standalone sequential models (SASRec, BERT4Rec) and Zero-shot LLMs. Under matched parameter budgets, REPREC matches or exceeds the performance of LoRA-FT and LLaRA across all datasets and LLM backbones (LLaMA and Qwen).
• User Regime Analysis:
  • Casual and Core Users: REPREC demonstrates significant gains over LoRA for users with short interaction histories (0–20 interactions). The injected user representation provides valuable collaborative signals that the LLM lacks in low-data regimes.
  • Power Users: For users with long histories (>20 interactions), the performance gap between REPREC and LoRA narrows, with REPREC performing comparably.
• Efficiency and Training/Inference Trade-offs:
  • Training Speed: Training REPREC is significantly faster than LoRA.
  • Short-Long Generalization: The authors investigated training with short prompt histories ($\ell=10$) and evaluating with longer contexts ($\ell=50$). While training on short histories degrades performance for power users, a hybrid approach (training short, testing long) recovers 85–100% of LoRA's performance for power users while reducing per-epoch training time by an average of 1.51×.
• Ablation: The study confirms that simply prepending untrained soft tokens (Frozen Injector) is insufficient; the learned alignment is essential. Furthermore, combining soft tokens with textual interaction history yields better results than soft tokens alone.

Significance and Claims

The paper claims that REPREC establishes a simple, modular, and practical approach for integrating collaborative signals into LLM-based recommendation systems. Its primary significance lies in demonstrating that high-quality sequential recommendation does not require complex architectural changes or expensive LLM fine-tuning.

By decoupling the collaborative signal extraction (frozen encoder) from the language reasoning (frozen LLM) via a lightweight alignment layer, REPREC offers a production-friendly pipeline that balances recommendation quality with computational efficiency. The authors emphasize that this approach is particularly effective for the "long tail" of users (casual and core) who constitute the majority of user bases in real-world systems, where data sparsity is a common challenge. The work suggests that user-level representation alignment is a viable alternative to backbone adaptation for scaling LLMs in recommendation scenarios.