Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE

1. Problem Statement

The paper addresses a critical limitation in applying Direct Preference Optimization (DPO)—a technique successful in Large Language Models (LLMs)—to Multimodal Sequential Recommendation (MSR) systems.

• The Core Issue: The "False Negative" Dilemma.
  • In NLP, negative samples (e.g., incorrect model outputs) are strictly undesirable.
  • In recommendation systems with implicit feedback, unobserved items (items a user didn't click) are not necessarily disliked; they are often just unexposed or unobserved positives.
  • Naïve DPO Failure: Standard DPO strategies often use Hard Negative Mining (selecting the item with the highest predicted score among non-targets). In MSR, this frequently selects "False Negatives" (items the user actually likes but hasn't clicked yet). Aggressively penalizing these items introduces erroneous gradients, distorts the ranking space, and leads to performance degradation.
• The Gap: Existing work lacks a systematic analysis of how negative selection strategies interact with DPO in the presence of implicit feedback, often confounding architectural complexity with the core alignment mechanism.

2. Methodology: RoDPO Framework

The authors propose RoDPO (Robust Direct Preference Optimization), a framework designed to align multimodal sequential recommenders with user preferences while mitigating the false negative problem.

A. Stochastic Top-K Negative Sampling (Core Innovation)

Instead of deterministically selecting a single "hardest" negative (argmax), RoDPO introduces a stochastic sampling mechanism:

1. Candidate Pool Construction: For a given context, the model identifies the top-$K$ highest-scoring items (excluding the ground-truth positive item $y_w$).
2. Stochastic Selection: The negative item ($y_l$) is sampled uniformly at random from this dynamic top-$K$ pool.
3. Mechanism: This approach retains the informative "hardness" of the signal (since the pool contains high-scoring candidates) but introduces controlled stochasticity. This prevents the model from repeatedly penalizing the same specific false negative, thereby smoothing optimization and reducing gradient noise.

B. Architecture & Components

• Multimodal Encoder: Uses modality-specific Transformers (for ID, text, and image features) fused via a weighted sum.
• Sparse Mixture-of-Experts (MoE): An optional component added to the encoder to scale model capacity efficiently. It uses a router to select a subset of experts (e.g., top-2 out of 4) for each input, allowing the model to handle complex preference boundaries without significantly increasing inference latency.
• Two-Stage Training Protocol:
  1. Warm-up (Stage 1): Standard Cross-Entropy (SFT) training to establish a reliable reference policy ($\pi_{ref}$).
  2. Alignment (Stage 2): Optimization of the joint objective $L_{total} = L_{CE} + \lambda L_{DPO}$, where the reference policy is frozen.

C. Objective Function

The DPO loss is computed directly using logit margins to avoid the need for an explicit reward model:
$$L_{DPO} = -\log \sigma \left( \beta [ (s_\theta(x, y_w) - s_\theta(x, y_l)) - (s_{ref}(x, y_w) - s_{ref}(x, y_l)) ] \right)$$
Where $y_l$ is the stochastically sampled negative from the top-$K$ pool.

3. Key Contributions

1. Identification of the False Negative Dilemma: The paper explicitly identifies that the brittleness of DPO in recommendation stems from the misalignment between "hard negative mining" and the nature of implicit feedback (unobserved positives).
2. Stochastic Top-K Negative Sampling: A simple, plug-and-play modification that balances gradient intensity with false-negative tolerance. It is shown to be the critical factor for DPO success in RecSys, more so than architectural complexity.
3. RoDPO Framework: A complete system integrating the sampling strategy with a Sparse MoE encoder for efficient capacity scaling.
4. Empirical Validation: Extensive experiments demonstrating that the proposed method outperforms state-of-the-art baselines with negligible inference overhead.

4. Experimental Results

The method was evaluated on three Amazon benchmark datasets: Toys and Games, Beauty, and Home and Kitchen.

• Performance Gains: RoDPO consistently outperformed strong baselines (including HM4SR, IISAN, and M3SRec).
  • Achieved up to +5.25% improvement in NDCG@5 and +7.67% in MRR@5 on the Toys and Games dataset compared to the best baseline.
  • Gains were consistent across different cutoff values ($K=5, 10$).
• Ablation Studies:
  • Negative Sampling: Stochastic Top-K sampling significantly outperformed both Random Sampling and deterministic Hard (Argmax) Sampling. Hard sampling alone showed only marginal gains, confirming the risk of false negatives.
  • Components: Removing the DPO objective caused a performance drop, confirming the necessity of preference alignment. Removing the Sparse MoE also reduced performance, indicating the value of increased capacity for fine-grained preference modeling.
• Efficiency:
  • Training: Negligible overhead compared to baselines (9.84s vs 9.81s per epoch) because the reference model shares the backbone and is frozen.
  • Inference: Nearly identical latency to baselines (0.45s vs 0.42s per batch) because the reference model is discarded during deployment, and the MoE layer activates only a subset of experts.

5. Significance and Conclusion

• Paradigm Shift: The paper demonstrates that DPO can be successfully adapted to recommendation systems, provided the negative sampling strategy is robust to the "false negative" nature of implicit feedback.
• Practical Impact: It offers a "plug-and-play" solution that improves ranking quality without requiring massive increases in computational cost or inference latency.
• Future Directions: The work paves the way for applying preference alignment techniques to larger-scale foundation models in recommendation, moving beyond simple likelihood estimation to explicit preference ranking.

In summary, RoDPO solves the specific domain mismatch between NLP-based DPO and Recommendation Systems by replacing deterministic hard negatives with stochastic top-K sampling, resulting in a more robust, accurate, and efficient multimodal recommender.