Technical Summary: On the Modality Gap and the Contrastive Loss in Multi-modal Representation Learning

Problem Statement

The paper investigates the modality gap observed in CLIP-style dual-encoder contrastive learning models. Despite being trained in a shared representation space, image and text embeddings often occupy distinct, non-overlapping regions (submanifolds) rather than forming a fully mixed distribution. While previous literature attributes this phenomenon to factors such as initialization biases, data heterogeneity, dimensionality, or temperature settings, the authors argue that the gap is fundamentally induced by a failure of the InfoNCE (Information Noise-Contrastive Estimation) formulation when applied to independent encoders.

The authors hypothesize that in a multi-modal setting with independent encoders, the uniformity term in the InfoNCE objective acts not merely as an information preservation mechanism but as a "degenerative term" with a divergence objective. Specifically, the loss function can be minimized by pushing the two modalities far apart while maintaining the correct ranking of positive pairs, leading to a local minimum where modalities are separated, particularly at low temperatures.

Methodology

Theoretical Analysis

The authors provide a theoretical analysis of the InfoNCE loss limit as the temperature parameter $\tau \to 0^+$. They demonstrate that for independent encoders, minimizing the loss at low temperatures encourages the negative similarities ($a_{ik}$) to converge toward their mean ($m_i$) while keeping the ranking of pairs intact. This results in a "narrow similarity distribution" where all negative samples from the other modality form a tight cluster distant from the anchor, creating the modality gap. They identify this as a mode-failure of the InfoNCE objective under low-temperature conditions.

Proposed Method: xNCE

To address this, the authors propose xNCE (Mixed Noise-Contrastive Estimation), a simple modification to the standard InfoNCE loss.

• Mechanism: Instead of sampling negative pairs exclusively from the opposite modality (inter-modal), xNCE mixes intra-modality negatives (e.g., image-image) and inter-modality negatives (image-text) within the denominator of the softmax probability.
• Rationale: By including intra-modality negatives, the loss function prevents the trivial solution of separating the two modalities. If the modalities drift apart, the intra-modality negative pairs would also become dissimilar, increasing the loss. This forces the encoders to align the modalities to satisfy the uniformity constraint across the mixed set.
• Implementation: A mixing ratio ($r_{mix}$) controls the proportion of intra- vs. inter-modal negatives. The loss remains a symmetric cross-entropy over the mixed similarity matrix.

Experimental Setup

The study conducts experiments in two settings:

1. Uni-modal (Image-to-Image): Using MNIST/EMNIST with two independent, identically initialized encoders (no weight sharing) to isolate the effect of the loss formulation from initialization or data idiosyncrasies.
2. Multi-modal (Image-to-Text): Fine-tuning CLIP (ViT-B/16) on the MS-COCO dataset.
   • Baselines: Standard InfoNCE (with learnable and fixed temperatures), Regularized InfoNCE (adding explicit alignment/uniformity terms), and high-temperature InfoNCE.
   • Evaluation: Linear probes for modality separability and downstream utility (digit classification, object detection mAP), retrieval performance (R@1, R@5, R@10), and zero-shot classification on CIFAR-10/100, Caltech-101, and ImageNet.

Key Results

1. Uni-modal Validation (MNIST)

• Gap Induction: Even with identical initialization and independent encoders, InfoNCE at low temperatures ($\tau=0.01$) actively generates a modality gap (modality distance $\approx 0.75$, 100% modality classification accuracy).
• xNCE Efficacy: xNCE significantly reduces this gap across all temperatures. At $\tau=0.01$, the modality distance drops to $0.09$, and modality classification accuracy falls below 100%, indicating successful alignment.
• Temperature Sensitivity: InfoNCE shows a sharp transition in modality overlap near $\tau \approx 0.09$. xNCE exhibits a more gradual transition and consistently lower modality distances.

2. Multi-modal Performance (MS-COCO)

• Modality Gap Reduction: Standard InfoNCE (learnable $\tau$) yields a large gap (distance $0.88$). Regularized InfoNCE reduces this to $0.12$ but incurs a ~25-30% training overhead. xNCE reduces the gap to $0.12$ (at $r_{mix}=0.01$) and $0.05$ (at $r_{mix}=0.50$) without additional loss terms or significant computational cost.
• Retrieval Performance:
  • Standard InfoNCE achieves the highest R@1 retrieval scores.
  • xNCE with small mixing ratios ($r_{mix}=0.01, 0.05$) maintains near-parity with InfoNCE on R@5 and R@10, with only a marginal drop in R@1.
  • High-temperature InfoNCE drastically reduces the gap but severely degrades retrieval performance (R@1 drops to $\approx 15\%$).
• Zero-Shot Classification:
  • Crucial Finding: xNCE with small mixing ($r_{mix}=0.01$) improves zero-shot Top-1 accuracy across all four benchmarks (CIFAR-10, CIFAR-100, Caltech-101, ImageNet) compared to the InfoNCE baseline.
  • In contrast, Regularized InfoNCE and high-temperature InfoNCE both fail to improve (and often degrade) zero-shot performance relative to the baseline, suggesting that explicit regularization or high temperatures damage the discriminative geometry required for transfer.
• Trade-off: Increasing the mixing ratio ($r_{mix}$) further reduces the gap and increases instance-level semanticity (mAP) but monotonically degrades zero-shot accuracy and retrieval precision, indicating a tunable trade-off between modality separability and discriminative structure.

Significance and Claims

The paper claims that the modality gap is not an inevitable byproduct of data or initialization but a mode-failure of the InfoNCE objective when used with independent encoders at low temperatures.

• Mechanistic Insight: The authors demonstrate that the gap arises because the uniformity term in InfoNCE can be satisfied by diverging modalities. xNCE corrects this by introducing intra-modality negatives, effectively removing the "divergence objective."
• Practical Advantage: Unlike high-temperature training or adding complex regularization terms, xNCE offers a simple, computationally efficient modification that reduces the modality gap without sacrificing (and in some cases, improving) the discriminative geometry necessary for zero-shot transfer and retrieval.
• Tunability: The method provides a controllable trade-off between modality alignment and within-modality semanticity, allowing practitioners to tune the mixing ratio ($r_{mix}$) based on specific downstream requirements (e.g., strict top-1 retrieval vs. modality-invariant representations).

The authors conclude that xNCE successfully mitigates the modality gap while preserving or enhancing the transfer capabilities of multi-modal models, challenging the necessity of complex regularization strategies or high-temperature regimes to achieve alignment.