Technical Summary: Eliminating Out-of-Domain Recommendations in LLM-based Recommender Systems

1. Problem Statement

Large Language Models (LLMs) are increasingly applied to conversational recommender systems due to their superior reasoning and instruction-following capabilities. However, a critical failure mode in current LLM-based recommenders is the generation of Out-of-Domain (OOD) items. Unlike traditional ID-based systems that strictly operate within a known catalog, LLMs often hallucinate items that do not exist in the target domain, leading to negative real-world impacts.

Existing approaches to mitigate OOD recommendations generally fall into three isolated paradigms:

1. Retrieval-based: Mapping user/item info to embeddings to retrieve in-domain items.
2. Constrained Generation: Restricting decoding to a catalog-dependent subspace (e.g., via prefix trees over item titles).
3. Item-Tokenizers: Mapping items to discrete token sequences and generating over this learned space.

These methods are typically developed with different backbones and prompts, making it difficult to systematically compare their relative strengths or understand the trade-offs between retrieval, generation, and tokenization in a unified setting.

2. Methodology: The RecLM Framework

The authors propose RecLM, a unified framework that bridges retrieval and generation under a single architecture and training protocol. The core innovation is the use of special control tokens to demarcate recommendation segments, allowing different grounding mechanisms to be plugged into the same decoding process.

2.1 Special Item Indicator Tokens

RecLM equips the backbone LLM (Llama3-8B-Instruct) with two special tokens:

• <SOI> (Start-of-Item): Signals the model is entering a recommendation segment where grounding constraints apply.
• <EOI> (End-of-Item): Marks the termination of the segment, after which the model resumes general text generation.

Between <SOI> and <EOI>, the framework delegates to one of three variants:

A. RecLM-ret (Retrieval-based)

• Mechanism: Uses the hidden state at <SOI> to retrieve the nearest item from a pre-built embedding index (constructed using BGE-M3).
• Training: Optimized with a standard language modeling loss plus an auxiliary retrieval loss that aligns the <SOI> hidden state with the ground-truth item embedding.

B. RecLM-cgen (Constrained Generation over Rewritten Titles)

• Mechanism: Generates item titles under a prefix-tree constraint built over a catalog of rewritten titles.
• Title Rewriter (TR): A lightweight module trained via Group Relative Policy Optimization (GRPO) to transform verbose item metadata into concise, human-readable titles. The TR is optimized using five reward components:
  • Item-to-Item Similarity (I2I): Preserves neighborhood structure.
  • User-to-Item Alignment (U2I): Ensures titles reflect user preferences.
  • Decoding Complexity (DC): Minimizes conditional perplexity.
  • Conciseness (CR): Favors shorter titles.
  • Discriminative Power (DPR): Ensures titles are distinguishable from similar items.
• Training: Employs Scope Mask Training, where the softmax denominator during loss calculation is restricted to tokens valid in the prefix tree, ensuring training matches inference constraints.

C. RecLM-token (Discrete Item-Token Generation)

• Mechanism: Maps each catalog item to a short sequence of discrete tokens (e.g., <a_11><b_2>...) using an RQ-VAE (Residual Vector Quantization) encoder. The LLM generates these token sequences between <SOI> and <EOI>.
• Training: Involves supervised fine-tuning to align the LLM with the codebook, followed by GRPO-based reinforcement learning. The reward function combines position-sensitive ranking rewards and inclusion-based rewards.

2.2 Multi-Round Dialogue Training

To prevent the model from becoming a rigid QA tool, the authors incorporate multi-round dialogue samples (combining ShareGPT dialogues with recommendation tasks) into the training data, ensuring the model can interleave recommendations naturally with general conversation.

3. Key Contributions

1. Unified Framework: RecLM unifies retrieval-based, constrained generation, and item-tokenization paradigms under a single architecture using the <SOI>/<EOI> interface, enabling fair comparison across distinct grounding strategies.
2. Lightweight Enhancement Modules: Introduction of a GRPO-trained Title Rewriter to compress metadata and a Scope Mask training strategy to align constrained decoding with training objectives.
3. Systematic Evaluation: A comprehensive empirical study across three public datasets (Steam, Amazon Movies, Amazon Toys) demonstrating that all variants strictly eliminate OOD recommendations while achieving competitive or state-of-the-art accuracy.

4. Experimental Results

Experiments were conducted on Steam, Amazon Movies, and Amazon Toys datasets, comparing RecLM against 12 baselines (traditional ID-based, frozen LLMs, fine-tuned LLMs, and item-tokenizer LLMs).

• OOD Elimination: All three RecLM variants achieved OOD@10 = 0, strictly eliminating out-of-domain recommendations across all datasets. This contrasts sharply with frozen LLMs (e.g., GPT-4o, Llama3) which showed OOD rates ranging from ~15% to over 90%.
• Accuracy: The constrained generation variants (RecLM-cgen and RecLM-token) achieved state-of-the-art performance, outperforming strong ID-based baselines (like SASRec) and other fine-tuned LLMs (like BIGRec, CtrlRec) in HR@10 and NDCG@10.
  • On the Steam dataset, RecLM-cgen achieved an HR@10 of 0.0868, a 14.8% improvement over the best non-RecLM baseline.
  • On the Toys dataset, RecLM-token achieved the highest NDCG@5 (0.0643), a 19.1% improvement.
• Control Reliability: In multi-turn dialogue tests, RecLM-cgen and RecLM-token achieved near-perfect control symbol scores (CSN), correctly generating exactly 10 <SOI> tokens for recommendation turns and zero for reasoning turns, without sacrificing general reasoning capabilities (MMLU, GSM8K).
• Ablation Studies: Incremental improvements were observed with each component: constrained generation improved ranking over unconstrained fine-tuning; scope masking further boosted accuracy; and the Title Rewriter (especially with full RL rewards) provided the final performance gains.

5. Significance and Claims

The paper claims that RecLM provides a systematic basis for comparing distinct paradigms to reduce item hallucinations. By integrating retrieval, constrained generation, and tokenization into a unified framework, the authors demonstrate that:

• Reliability and Quality are not mutually exclusive: It is possible to strictly enforce catalog fidelity (OOD@10 = 0) while simultaneously achieving state-of-the-art recommendation accuracy.
• Paradigm Trade-offs: The unified view reveals complementary strengths; while retrieval (RecLM-ret) is effective, constrained generation (RecLM-cgen/token) offers superior ranking accuracy and conversational flexibility.
• Practical Utility: The framework offers a practical tool for designing LLM-based recommendation systems in both academic and industrial settings, providing a clear path to mitigate the OOD problem that plagues current generative recommenders.

The authors acknowledge limitations regarding inference latency due to the autoregressive nature of LLMs and the need for future work on diversity, fairness, and long-term user satisfaction metrics. However, they position RecLM as a foundational step toward reliable, hallucination-free LLM recommenders.