Multimodal Representation Learning Conditioned on Semantic Relations

Technical Summary: Multimodal Representation Learning Conditioned on Semantic Relations

Problem Statement
Current multimodal representation learning, dominated by contrastive models like CLIP, typically generates a single, relation-agnostic embedding for each sample. While effective for general-purpose tasks, this approach assumes that a sample's relevance is static across all contexts. However, in real-world applications (e.g., e-commerce recommendations or information retrieval), the relevance between two samples is inherently relation-dependent. Different semantic relations emphasize different aspects of multimodal data; for instance, a nursing pillow and a bottle sterilizer may share little visual or textual similarity but are highly related under the semantic relation of "infant feeding interests." Existing methods either treat relations as implicit prompt modifications (leading to uncontrolled variations) or rely on task-specific heads that lack reusability. There is a lack of frameworks that treat semantic relations as explicit conditions of the representation learning process itself.

Methodology: Relation-Conditioned Multimodal Learning (Rcml)
The authors propose Rcml, a framework that integrates semantic relations directly into the representation learning process. The methodology consists of three core components:

1. Relation-Conditioned Contrastive Learning:
   Unlike standard contrastive learning which aligns only matched image-text pairs (diagonal positives), Rcml enables many-to-many alignment across samples based on semantic relations. The framework defines a unified contrastive loss that jointly optimizes:

   • Cross-modal alignment: Aligning text and image representations under the same relation.
   • Intra-modal coherence: Ensuring consistency among samples connected by the same relation within a single modality (text-text or image-image).
     The loss function is defined as:
     $$\mathcal{L} = \frac{1}{2}(\mathcal{L}_{txt-img} + \mathcal{L}_{img-txt}) + \lambda(\mathcal{L}_{txt-txt} + \mathcal{L}_{img-img})$$
     where positive pairs are defined by the specific semantic relation $e_{ij}$ rather than global identity.

2. Relation-Conditioned Pair Construction:
   The framework constructs training pairs based on two types of relations:

   • Intra-sample relations: Aligning the text and image of the same sample.
   • Inter-sample relations: Constructed from user behavior (e.g., co-purchase or co-read histories). Users are clustered based on interaction patterns, and samples frequently co-consumed by a cluster are linked. These links are assigned natural-language descriptions (e.g., "Co-purchased by users with [cluster-specific] preferences") that serve as the conditioning signal.

3. Relation-Conditioned Representation Module:
   Using a CLIP backbone, Rcml introduces a module that adapts embeddings to specific semantic contexts. Instead of a static global pooling, the model employs a relation-guided attention mechanism.

   • The semantic relation description $e_{ij}$ is encoded into a global embedding $h^E_{ij}$.
   • This embedding guides an attention mechanism to aggregate token-level representations ($H^i_x$) into a relation-conditioned representation $z^i_x(e_{ij})$.
   • The attention weight is a balance between relation-aware contextual attention (driven by $h^E_{ij}$) and undirected alignment (preserving global consistency via special tokens like [CLS] or [EOT]). A coefficient $\beta$ controls this trade-off.

Key Contributions

1. Rcml Framework: The introduction of a multimodal learning framework where semantic relations act as explicit conditions, moving beyond single relation-agnostic embeddings to a family of context-dependent embeddings.
2. Many-to-Many Contrastive Learning: Extending contrastive learning beyond isolated image-text pairs to model complex inter-sample dependencies under semantic relations.
3. Empirical Validation: Demonstrating consistent improvements over strong baselines (CLIP, DeCLIP, UniCL, SigLIP, ImageBind) on retrieval and classification tasks across multiple datasets and settings (zero-shot, fine-tuned, and out-of-domain).

Experimental Results
The authors evaluated Rcml on the Amazon Product dataset (across 7 domains: Electronics, Automotive, Office, Baby, Pet, Music, Sports) and the Goodreads dataset.

• Relation-Conditioned Retrieval: Rcml significantly outperformed all baselines, achieving the best Hit@5 results on 36 out of 40 metrics. It improved overall Hit@5 by approximately 30.79% compared to standard CLIP. Notably, it generalized well to the out-of-domain Goodreads dataset.
• Relation Type Prediction: The model achieved high Top-3 accuracy in predicting the semantic relation connecting a product pair, exceeding random baselines by substantial margins (30% on Amazon, 37.5% on Goodreads).
• Relation Validity Prediction: In a supervised binary classification task, Rcml demonstrated robustness and superior performance in validating specific relation instances compared to models lacking explicit relation-aware mechanisms.
• Efficiency: Despite computing relation-conditioned embeddings, Rcml remained efficient (14.32 ms/sample) with a parameter count (152.33M) comparable to CLIP, significantly outperforming the much larger ImageBind (1200M+ parameters) which still underperformed on most tasks.
• Ablation Studies: Removing inter-sample relations caused the largest performance drop (38.68%), confirming the necessity of many-to-many learning. Removing the semantic text of relations also caused a significant decline (11.60%), highlighting the value of contextual guidance.

Significance and Claims
The paper claims that Rcml provides a general and extensible framework for relation-aware multimodal representation learning. By treating semantic relations as explicit conditions rather than auxiliary signals, the framework learns embedding spaces that are more structured and flexible, organizing samples according to the specific relational context in which they are compared. The authors assert that this approach captures functional and intent-driven associations that surface-level similarity models miss, offering a more effective solution for downstream tasks where relevance is context-dependent. The work suggests that future multimodal systems should move beyond static, single-embedding paradigms toward dynamic, relation-conditioned representations.