Technical Summary: SynGR – Unleashing the Potential of Cross-Modal Synergy for Generative Recommendation

1. Problem Statement

Generative Recommendation (GR) has emerged as a paradigm that formulates item recommendation as a sequence-to-sequence generation task using discrete semantic identifiers (e.g., quantized tokens). While recent studies have integrated multimodal signals (visual and textual) to enrich item evidence, existing approaches predominantly rely on alignment-centric fusion. These methods enforce cross-modal consistency through shared representations, effectively capturing redundant information ($R$) and modality-specific unique information ($U$), but failing to exploit synergistic information ($S$).

Synergistic information refers to emergent item properties that arise only from the joint interplay of modalities and cannot be inferred from any single modality alone (e.g., combining visual texture with textual price to infer "luxury positioning"). The paper identifies a "synergy gap" in current GR models:

• Modality Imbalance: Textual inputs are often discrete and semantically compact, while visual embeddings are continuous and noisy. This disparity leads to a mismatch in information density.
• Unimodal Shortcuts: Under the generative training objective, models tend to follow the "path of least resistance," relying heavily on the dominant modality (typically text) to minimize generation loss.
• Suppression of Synergy: This unimodal convergence suppresses the high-order interactions required to capture synergistic semantics, limiting the model to surface-level feature matching.

2. Methodology: The SynGR Framework

To address these limitations, the authors propose SynGR, a framework designed to explicitly disrupt unimodal shortcuts and compel the model to leverage cross-modal dependencies. The framework consists of three core components:

A. Multimodal Tokenization (RQ-VAE)

Continuous visual and textual features are discretized into a unified vocabulary using a Residual-Quantized Variational AutoEncoder (RQ-VAE).

• Mechanism: Items are encoded into hierarchical semantic IDs via multi-level residual quantization.
• Unified Stream: Textual tokens are assigned lowercase prefixes, and visual tokens use uppercase prefixes, allowing the generative backbone to process multimodal content as a single, coherent sequence while preserving modality identity.

B. Saliency-Aware Masking Mechanism

To satisfy the theoretical requirement of "Unimodal Obscuration" (preventing reliance on single streams), SynGR employs an adaptive masking strategy during training:

• Diagnosis: The model analyzes self-attention weights from the final encoder layer to compute a global saliency score for each token.
• Identification: It identifies the dominant modality (text or vision) based on average saliency density.
• Intervention: The top-$r$ most salient tokens within the dominant modality are masked (replaced with [MASK]). This forces the generator to bridge information gaps through cross-modal reasoning rather than isolated cues.

C. Synergistic Contrastive Learning

To further isolate and refine synergistic semantics, the framework introduces a contrastive objective that regularizes the latent space:

• Triplet Construction: For each history, three views are constructed:
  1. Anchor ($H_{mask}$): The sequence with dominant modality tokens masked.
  2. Positive ($H_{ori}$): The original, holistic multimodal sequence.
  3. Negative ($H_{uni}$): A unimodal sequence (shortcut view) derived from a single modality.
• Objective: The model is trained to pull the representation of the masked anchor closer to the holistic positive view while pushing it away from the unimodal negative view. This explicitly encourages the preservation of synergistic information ($S$) while suppressing redundant ($R$) and unique ($U$) shortcuts.

D. Unified Optimization

The final loss function combines the negative log-likelihood (NLL) for next-token prediction across all views with the synergistic contrastive loss ($L_{Syn}$):
$$L = L_{Gen} + \lambda L_{Syn}$$
This ensures the model maintains fundamental unimodal semantics while learning to capture complementary cross-modal interactions.

3. Key Contributions

1. Insight: The paper identifies a "synergy gap" in generative recommendation, demonstrating that information density mismatches induce unimodal shortcuts that suppress higher-order cross-modal semantics.
2. Method: The proposal of SynGR, a framework that actively disrupts unimodal shortcuts via saliency-aware masking and synergistic contrastive learning to force the exploitation of cross-modal dependencies.
3. Evaluation: Extensive experiments across three real-world datasets (Arts, Games, Instruments) demonstrate that SynGR consistently outperforms strong baselines, achieving average gains of approximately 10% over state-of-the-art methods.

4. Experimental Results

• Performance: SynGR achieved the best performance across all metrics (HR@K, NDCG@K) on the Arts, Games, and Instruments datasets. Notably, on the Instruments dataset, it surpassed the strongest baseline (MACRec) by 28.58% in HR@10 and 12.17% in NDCG@10.
• Efficiency: Despite multi-view training, SynGR demonstrated superior training efficiency, reducing training time by ~23% on the Arts dataset compared to MACRec, due to parallel processing and lightweight masking mechanisms.
• Ablation Studies: Removing the saliency masking (w/o SM), unimodal negative construction (w/o UN), or the contrastive objective (w/o SCL) resulted in significant performance degradation, confirming the necessity of each component.
• Case Study: In a qualitative example involving World Cup memorabilia, SynGR correctly ranked a "Messi World Cup Winner Poster" as the top recommendation by integrating specific textual and visual cues, whereas the baseline MACRec favored items with only coarse visual or textual similarity.

5. Significance and Claims

The paper claims that SynGR represents a significant step forward in multimodal Generative Recommendation by shifting the focus from mere alignment to synergistic interaction.

• Theoretical Contribution: It provides an information-theoretic analysis (using Partial Information Decomposition) showing that existing objectives fail to capture the synergistic component $S$, which is critical for capturing emergent item properties.
• Practical Impact: By preventing models from collapsing into unimodal shortcuts, SynGR enables a deeper understanding of user intent that goes beyond surface-level feature matching.
• Future Directions: The authors modestly note that while SynGR focuses on item-side synergistic semantics, future work could incorporate interaction-driven collaborative signals and extend the paradigm to richer modality combinations (e.g., audio and video).

The work is supported by the National Key Research and Development Program of China and the National Natural Science Foundation of China.