Technical Summary: MEMOIR – Temporal Behavioral Memory for Recommendation Across the Preference-Drift Spectrum

1. Problem Statement

Existing sequential recommendation models (e.g., SASRec, BERT4Rec) and recent LLM-enhanced methods typically treat user preferences as either static summaries or byproducts of next-item prediction. They implicitly learn behavioral patterns but fail to explicitly model preference evolution. The authors argue that two users with identical current purchase histories may have vastly different needs if their trajectories differ (e.g., a long-time fitness enthusiast vs. a novice).

Empirical analysis of the Amazon Reviews 2023 dataset reveals that preference drift is pervasive:

• The mean Jensen-Shannon divergence between adjacent monthly category distributions is 0.795 (out of 1.0).
• 92.2% of items interacted with in any given window are new to the user.
• Even users in the bottom quartile of drift scores exhibit substantial change.

Current LLM-enhanced approaches often use LLMs as offline preprocessing steps to generate summaries or augmented views for ID-based backbones. These semantic signals are typically discarded at inference time, and the core architecture remains ID-based, failing to carry the structure of preference evolution into the final recommendation score.

2. Methodology: The MEMOIR Framework

MEMOIR (Memory-based Evolution MOdel for Inference in Recommendation) inverts the standard relationship: the LLM acts as a first-class temporal memory encoder, and contrastive learning encodes the structure of evolution directly into the representation.

2.1 Architecture Components

1. Temporal Segmentation & Memory Encoding:

   • User interaction histories are partitioned into temporal windows based on calendar months.
   • Each window is serialized into a natural language description (e.g., "User highly rated 'running shoes'...").
   • An LLM encoder (TinyLlama-1.1B-Chat with LoRA fine-tuning) maps these descriptions to semantic memory embeddings ($m_t$).
   • This yields a sequence of behavioral memories $M_u = [m_1, \dots, m_W]$.

2. Evolution-Preserving Contrastive Loss ($L_{evo}$):

   • Temporal Smoothness ($L_{smooth}$): Enforces continuity between adjacent windows using an InfoNCE-style loss, treating adjacent windows of the same user as positive pairs.
   • Directional Consistency ($L_{dir}$): Enforces that the direction of evolution remains coherent. It computes direction vectors ($d_t = \hat{m}_{t+1} - \hat{m}_t$) and penalizes misalignment between consecutive direction vectors. This encodes the inductive bias that preference shifts (e.g., casual to sportswear) should follow a consistent trajectory.

3. Trajectory Prediction:

   • A GRU processes the memory sequence to produce a final hidden state.
   • An MLP projects this state to predict the next-step memory ($\hat{m}_{W+1}$).
   • A trajectory extrapolation loss ($L_{extrap}$) supervises this prediction using a hold-out last window.

4. Evolution-Aware Aggregation:

   • Instead of a simple mean, the final user representation ($u^*$) is an MLP fusion of three signals:
     • Current State: Recency-weighted aggregate of memories ($\bar{m}$).
     • Direction: The evolution direction vector ($d$).
     • Predicted Future: The extrapolated next-step memory ($\hat{m}_{W+1}$).
   • This ensures users with identical current states but different trajectories receive distinct representations.

5. Behavioral Consistency Loss ($L_{con}$):

   • A regularizer that penalizes the distance between the normalized aggregate memory and the normalized target item embedding. This anchors the semantic trajectory to the actual item embedding space, preventing drift into semantically detached regions.

2.2 Training Objective

The total loss combines recommendation, evolution, consistency, and extrapolation:
$$L = L_{rec} + \lambda_1 L_{evo} + \lambda_2 L_{con} + \lambda_3 L_{extrap}$$
where $L_{rec}$ is an InfoNCE recommendation loss over user-item dot products.

3. Key Contributions

• Architectural Shift: Proposes a framework where the LLM is the primary user encoder for temporal windows, and contrastive learning is the mechanism for learning evolution structure, rather than an auxiliary offline step.
• Evolution-Aware Representation: Introduces a method to fuse current state, evolution direction, and predicted future into a single user embedding.
• Drift-Stratified Analysis: Moves beyond aggregate metrics to analyze performance across a spectrum of user preference drift, revealing that gains are concentrated in specific user segments.

4. Experimental Results

Experiments were conducted on the Electronics and Clothing_Shoes_and_Jewelry categories of Amazon Reviews 2023 (99,355 users).

4.1 Aggregate Performance

• MEMOIR achieved an NDCG@10 of 0.0643, statistically tied with the strongest baseline, UniSRec (0.0641).
• The models split the four reported metrics 2–2:
  • MEMOIR leads: NDCG@10 and MRR.
  • UniSRec leads: HR@10 and HR@20.
• Both significantly outperformed six other baselines (including SASRec, CL4SRec, and SRA-CL).
• SASRec-Text (SASRec with frozen semantic embeddings replacing ID embeddings) underperformed vanilla SASRec, indicating that semantic features alone do not improve recommendation without specific architectural alignment mechanisms.

4.2 Ablation Study

Surprisingly, no single architectural component (evolution-preserving contrastive loss, directional-consistency term, or temporal window segmentation) individually explained the majority of MEMOIR's gain over ID-based baselines.

• Removing any single component resulted in performance within 2% of the full model on aggregate NDCG@10.
• Even replacing pretrained item encoders with random ID embeddings ("w/ Random-Init Items") yielded the best NDCG@10 among ablation variants (0.0650), suggesting the aggregate gain is not driven by a single isolated module.

4.3 Drift-Stratified Analysis (The Core Finding)

When performance was stratified by a composite preference-drift score (High, Medium, Low):

• High and Low Drift Extremes: MEMOIR outperformed UniSRec on ranking-quality metrics (NDCG@10, MRR).
• Medium Drift (50% of users): UniSRec led on all four metrics.
• Conclusion: The aggregate "tie" masks a specific pattern: MEMOIR's advantage is concentrated at the extremes of the drift distribution, while UniSRec dominates the middle band and volume-oriented metrics (HR) across all strata.

5. Significance and Claims

The paper posits that its most substantive and reproducible finding is not the aggregate near-tie with UniSRec, nor the contribution of any single ablated component. Instead, the key insight is the drift-stratified performance pattern:

• MEMOIR is specifically effective for users at the high- and low-drift extremes of the distribution, where modeling the trajectory (direction and future) provides a ranking-quality advantage.
• The model's directional-consistency assumption (which assumes monotonic drift) may be less effective for the "medium-drift" users who exhibit more oscillating or seasonal preferences, explaining why UniSRec dominates that segment.

The authors conclude that while MEMOIR offers richer evolution-aware representations, it trades off the inference simplicity and sub-millisecond latency of ID-based approaches (like SASRec) for the computational cost of LLM-derived window embeddings. The paper leaves open the mechanistic explanation for why the gain concentrates at the drift extremes, identifying this as a direction for future work.