HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

1. Problem Statement

Current user modeling benchmarks and evaluation protocols suffer from significant limitations that hinder the development of robust, generalizable recommendation systems:

• Narrow Scope: Existing benchmarks (e.g., MovieLens, Amazon Reviews by category) focus on single-domain, short-session interactions, failing to capture the cross-domain and long-term nature of real-world user behavior.
• Evaluation Flaws: Standard evaluation methods (Leave-One-Out or Ratio-Based splits) often leak future data into training or fail to test for Out-of-Distribution (OOD) generalization. They assume the test distribution matches the training distribution, overestimating model robustness.
• Temporal Blindness: Most benchmarks do not evaluate a model's ability to reason over long time horizons (e.g., predicting behavior years after training data) or handle the temporal drift of user preferences.
• Data Accessibility: High-quality, large-scale datasets with rich temporal and cross-domain signals (like PinnerFormer's data) are often proprietary, limiting reproducibility.

2. Methodology: The HORIZON Benchmark

The authors introduce HORIZON, a fully open-source, large-scale benchmark designed to reformulate user modeling along three axes: Dataset, Task, and Evaluation.

A. Dataset Construction

• Source: Reformulated from the Amazon Reviews 2023 dataset.
• Scale: Contains 54.5 million users and 34.5 million unique items with 485.89 million interactions.
• Cross-Domain Unification: Unlike previous approaches that segment Amazon data by category, HORIZON merges all 33+ product categories into unified user histories. This forces models to learn cross-domain transitions (e.g., a user buying electronics and then books) rather than isolated domain-specific patterns.
• Temporal Depth: The dataset spans a long timeline, with a specific focus on interactions before and after the year 2020, enabling rigorous temporal extrapolation.

B. Proposed Evaluation Framework

HORIZON introduces a structured evaluation protocol that disentangles generalization across User Identity (Seen vs. Unseen) and Time (Aligned vs. Extrapolated). This creates four distinct evaluation settings for Task 1:

1. In-Distribution, Temporally Aligned (1a): Standard Leave-One-Out on users seen during training, restricted to a temporally consistent subset.
2. In-Distribution, Temporal Extrapolation (1b): Predicting future interactions for seen users occurring after the training cutoff (testing time generalization).
3. OOD-User, Temporally Aligned (1c): Predicting interactions for unseen users within the training time window (testing user generalization).
4. OOD-User, Temporal Extrapolation (1d): The most challenging setting: predicting future interactions for unseen users occurring after the training cutoff (testing both user and time generalization).

C. Task Formulations

The benchmark defines three specific tasks:

• Task 1: Next-Item Recommendation: Traditional sequential recommendation using ID-based models (GRU4Rec, SASRec, BERT4Rec, CORE).
• Task 2: LLM-Based Query Reformulation: Using Large Language Models (LLMs) to generate 10 diverse search queries based on user history to retrieve items via a vector index (BLAIR encoder + FAISS).
• Task 3: LLM-Based Long-Horizon Modeling: Using LLMs to generate natural language descriptions of the next 10 items a user will interact with, capturing long-term intent evolution.

3. Key Contributions

1. First Open-Source Cross-Domain Long-Horizon Benchmark: HORIZON is the first fully open benchmark supporting large-scale, cross-domain, and temporally grounded evaluation of sequential recommenders.
2. Unified Evaluation Protocol: It proposes a framework that explicitly separates seen/unseen users and temporal proximity, revealing performance gaps obscured by traditional in-distribution metrics.
3. Comprehensive Empirical Analysis: The paper provides a deep dive into the trade-offs between in-distribution accuracy, temporal generalization, and OOD user adaptation, highlighting the limitations of current architectures.

4. Experimental Results & Findings

A. Traditional ID-Based Models

• Performance Drop in OOD: Models (SASRec, BERT4Rec) perform well in In-Distribution settings but suffer significant degradation (often >50% drop in Recall/NDCG) when tested on Unseen Users (OOD).
• Temporal Extrapolation: Models struggle to generalize to future time periods, even for seen users. BERT4Rec, while state-of-the-art in standard settings, degrades significantly for OOD users but remains surprisingly competitive in long-range temporal extrapolation (Recall@50/100) compared to GRU-based models.
• Limitation of ID-Based Approaches: The reliance on item/user IDs without semantic grounding makes it difficult to adapt to new items (cold start) or evolving user preferences over time.

B. LLM-Based Approaches

• Query Reformulation (Task 2): LLMs (LLaMA-3.1, Qwen3, Gemma2) showed modest performance in retrieving relevant items via generated queries. Qwen3-8B performed best, but absolute Recall remained low (<4% at Recall@100), indicating difficulty in mapping semantic queries to specific catalog items.
• Long-Horizon Modeling (Task 3): LLMs struggled to predict specific future items over long horizons. While they could generate semantically plausible descriptions, the retrieval accuracy was low.
• Fine-Tuning vs. Zero-Shot: Fine-tuning LLMs (LoRA or Full Fine-Tuning) for next-item prediction did not significantly outperform zero-shot baselines. In fact, fine-tuned models often matched the low performance of zero-shot retrieval, suggesting that standard instruction tuning is insufficient for long-tail, discrete item prediction tasks.
• Reasoning Models: Scaling up to reasoning models (e.g., Qwen3-235B) did not yield substantial improvements over smaller models, suggesting that current scaling laws may not directly translate to user behavior modeling.

5. Significance and Implications

• Redefining Robustness: The paper demonstrates that high accuracy on standard benchmarks (like Leave-One-Out) is a poor proxy for real-world robustness. A model must be evaluated on its ability to handle distribution shifts in both time and user identity.
• Need for Semantic Grounding: The failure of ID-based models in OOD settings and the struggle of LLMs to map semantics to specific items highlight the need for hybrid architectures that combine collaborative filtering with rich semantic representations.
• Future Research Direction: HORIZON establishes a foundation for developing temporally robust, cross-domain, and general-purpose user models. It calls for new evaluation metrics that prioritize generalization over in-distribution accuracy and encourages research into better alignment between generative LLMs and recommendation objectives.

In conclusion, HORIZON exposes a critical gap between current state-of-the-art recommendation models and the demands of real-world deployment, providing the necessary infrastructure and evaluation standards to bridge this gap.