Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation

This paper proposes an efficient knowledge distillation framework that leverages pre-trained LLMs to generate textual user profiles for sequential recommenders, thereby enhancing user semantic understanding without incurring real-time LLM inference costs or requiring architectural modifications.

The Gist

Imagine you are a personal shopper for a massive department store. Your job is to guess what a customer will buy next based on what they've bought before.

The Problem: The "Old School" Shopper vs. The "Genius" Shopper

1. The Old School Shopper (Traditional Recommenders):
This is like a shop assistant who only looks at a customer's receipt.

• Scenario: "You bought shampoo, then conditioner, then a hairdryer. So, you probably want a hairbrush next."
• The Flaw: They are great at spotting patterns, but they don't understand why. They don't know if the customer loves "eco-friendly" products, hates "cruelty-free" items, or is just buying gifts for a friend. They are efficient and fast, but a bit shallow.

2. The Genius Shopper (Large Language Models - LLMs):
This is like a highly educated consultant who can read the customer's entire diary, social media, and reviews.

• Scenario: "Ah, I see you bought that organic face cream. You seem to value natural ingredients and have sensitive skin. You also rated that synthetic nail polish poorly because it smelled bad. You're a 'discerning beauty enthusiast' who prioritizes visible results."
• The Flaw: This genius is incredibly smart, but they are slow and expensive to hire. If you try to ask this consultant to make a recommendation for every single customer the moment they walk into the store, the line will stretch out the door, and the store will go bankrupt paying their hourly rate.

The Solution: The "Mentorship" Program (Knowledge Distillation)

The authors of this paper came up with a clever way to get the best of both worlds. They didn't try to hire the Genius Shopper to work the register. Instead, they set up a Mentorship Program.

Here is how it works, step-by-step:

Step 1: The Interview (Offline Phase)

Before the store opens, the "Genius Shopper" (the LLM) sits down with the store manager. The manager gives the Genius a list of 100,000 customers and their purchase histories.

• The Genius reads through them and writes a detailed personality profile for each customer (e.g., "User 405 is a budget-conscious tech geek who loves sci-fi movies").
• Crucial Point: This happens once, before the store opens. It's slow, but it's okay because it's offline.

Step 2: The Training (Distillation Phase)

Now, the "Old School Shopper" (the fast, traditional model) starts training.

• The manager shows the Old School Shopper a customer's purchase history.
• The Old School Shopper makes a guess.
• Then, the manager says, "No, look at the Genius's Profile for this customer. The Genius says this person loves 'organic skincare.' Try to make your internal 'brain' feel the same way about this customer."
• The Old School Shopper adjusts its internal settings to mimic the understanding of the Genius, without needing the Genius to be there. It learns to "think" like the Genius by studying the profiles the Genius wrote.

Step 3: The Grand Opening (Serving Phase)

The store opens!

• A customer walks up.
• The Old School Shopper is now working the register.
• Because of the training, when the customer buys a face mask, the Old School Shopper instantly thinks, "Ah, this is the 'organic skincare' person! I should recommend that new organic serum!"
• The Magic: The Old School Shopper is still fast (like a normal computer) and cheap to run, but it now has the wisdom of the Genius Shopper in its head.

Why is this a big deal?

1. Speed: You don't have to wait for the slow Genius to think during the transaction. The recommendation happens instantly.
2. Smarts: The recommendations are much better because they understand the person, not just the items.
3. No Re-invention: You don't have to rebuild the whole store or hire new staff. You just train the existing staff to think a bit deeper.

The Results

The paper tested this on four different types of "stores" (Beauty products, Movies, etc.).

• Accuracy: The trained "Old School" shoppers became significantly better at guessing what people wanted next (up to 23% better in some cases).
• Efficiency: They were still lightning-fast, whereas trying to use the "Genius" directly would have been 50 to 180 times slower.

In a nutshell: They taught a fast, simple robot to understand human feelings by having it study the notes written by a slow, super-smart human, so the robot can make smart decisions instantly without needing the human around.

Technical Summary

1. Problem Statement

Sequential Recommender Systems (SRS), such as SASRec and BERT4Rec, excel at modeling temporal user behavior but suffer from two primary limitations:

• Data Sparsity: They struggle to generalize when interaction data is sparse.
• Semantic Limitation: They rely heavily on interaction patterns (item IDs) and fail to capture rich user semantics (e.g., preferences for specific ingredients, styles, or genres) beyond the raw interaction history.

While Large Language Models (LLMs) offer superior semantic understanding, integrating them directly into recommendation pipelines creates prohibitive inference latency and computational costs, making them unsuitable for real-time, large-scale deployment. Existing distillation methods often focus on item-centric knowledge or require expensive fine-tuning of the LLM itself, failing to leverage user-specific semantics efficiently.

2. Methodology

The authors propose a two-phase knowledge distillation framework that transfers user-centric semantic knowledge from a pre-trained LLM into a standard Transformer-based sequential recommender without modifying the recommender's architecture or requiring LLM inference at serving time.

A. Offline User Profile Generation

1. Aggregation: For each user, textual metadata (titles, descriptions, categories) from their interaction history is aggregated.
2. LLM Inference: A pre-trained LLM (Gemma-2-9b) processes this text via a structured prompt to generate a comprehensive textual user profile. The prompt instructs the LLM to analyze patterns, identify preferences, distinguish high/low-rated items, and synthesize a character description.
3. Embedding: The generated text is encoded using a multilingual text encoder (E5-large) and projected into the same vector space as the recommender model using UMAP (Uniform Manifold Approximation and Projection) for dimensionality reduction. These embeddings are pre-computed and frozen.

B. Two-Stage Training Strategy

The sequential recommender (e.g., SASRec or BERT4Rec) is trained in two distinct stages:

1. Distillation Stage (Joint Training):

   • The model is optimized using a combined loss function:
     $$L = \alpha \cdot \beta \cdot L_{distill} + (1 - \alpha) \cdot L_{model}$$
   • $L_{model}$: Standard next-item prediction loss (e.g., Cross-Entropy).
   • $L_{distill}$: Mean Squared Error (MSE) between the model's internal user representation ($H_k$) and the frozen LLM-generated user profile embedding.
   • Dynamic Scaling ($\beta$): A per-batch scaling factor is introduced to balance the magnitude difference between the small distillation loss and the larger model loss. $\beta$ is computed as the ratio of the model loss to the distillation loss (with a stop-gradient operation) to prevent numerical domination.
   • Representation Alignment: The model aggregates hidden states from the transformer layers (using mean pooling or exponential weighting) to match the LLM profile embedding.

2. Fine-Tuning Stage:

   • The auxiliary distillation loss is removed.
   • The model is fine-tuned exclusively on the next-item prediction task ($L_{model}$) to refine its predictive capabilities while retaining the semantic knowledge embedded during the first stage.

3. Key Contributions

• User-Centric Distillation: Unlike prior work that distills item semantics, this method specifically targets user profile semantics, allowing the model to understand who the user is, not just what they clicked.
• Zero Inference Overhead: The LLM is used only during the offline training phase. At serving time, the system operates with the same efficiency and latency as a standard sequential model (no LLM inference required).
• Architecture Agnostic: The method does not require architectural changes to the sequential recommender nor fine-tuning of the pre-trained LLM.
• Dynamic Loss Balancing: The introduction of the dynamic scaling factor $\beta$ effectively manages the disparity in magnitude between reconstruction and prediction losses without manual hyperparameter tuning.

4. Experimental Results

The method was evaluated on four diverse datasets: Beauty (Product Reviews), ML-20M (Movies), Kion (Movies), and Amazon M2 (E-commerce).

• Performance Gains:

  • SASRec + Distillation: Showed consistent improvements across all datasets, with NDCG@10 gains ranging from 2.02% to 5.62%.
  • BERT4Rec + Distillation: Demonstrated significant gains, particularly on the Beauty dataset where Recall@10 improved by 23.53% and NDCG@10 by 19.61%.
  • Comparison with LLM Baselines: The proposed method outperformed IDGenRec (a state-of-the-art LLM-based baseline) on three out of four datasets (ML-20M, Kion, Amazon M2) and was competitive on Beauty, despite IDGenRec's heavy reliance on semantic ID generation.

• Efficiency Analysis:

  • Training: The distillation approach increased training time by only 5–25% compared to vanilla SASRec, whereas IDGenRec required 1.5–2.3× longer.
  • Inference: The proposed method matched the latency of vanilla SASRec. In contrast, IDGenRec was 50–180× slower due to beam search text generation during inference.

• Ablation Studies:

  • Dynamic scaling ($\beta$) was crucial for balancing losses, shifting the optimal static weight ($\alpha$) from 0.8 (without scaling) to 0.4 (with scaling).
  • Aligning with the final transformer layer yielded the best results.

5. Significance

This paper presents a practical and scalable pathway for integrating the semantic reasoning capabilities of LLMs into industrial recommendation systems. By decoupling the heavy semantic processing (LLM) from the real-time inference loop, the authors achieve a "best of both worlds" scenario:

1. Semantic Richness: The model learns deep user preferences and behavioral nuances from the LLM.
2. Operational Efficiency: The system maintains the low-latency, high-throughput requirements of production environments.

The work demonstrates that knowledge distillation is a viable strategy to overcome the "accuracy vs. efficiency" trade-off, offering a solution that is significantly faster than full LLM inference while delivering superior recommendation quality compared to traditional sequential models.