Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP

1. Problem Statement

Recent literature has proposed the Intra-Modal Misalignment Hypothesis, suggesting that Contrastive Language-Image Pre-training (CLIP) and similar models produce sub-optimal embeddings for image-only tasks (e.g., image retrieval, few-shot classification).

• The Hypothesis: Because CLIP is trained solely on an inter-modal (image-text) contrastive loss, it neglects intra-modal (image-image) alignment.
• The Consequence: Proponents argue this leads to poorly calibrated distances between images, where images of the same class may be further apart than images of different classes (high intra-class variance), and where image-text similarities are distributed differently than image-image similarities (the "modality gap").
• Current Mitigation: To address this, prior works have proposed avoiding direct image-image comparisons, instead converting images to text tokens (e.g., Optimization-based Textual Inversion) or using image-text similarities as a proxy for image-image relationships.

The Core Question: Is the observed performance gap in image-only tasks truly caused by a fundamental misalignment in the embedding space, or is it a misinterpretation of normal embedding behavior and task ambiguity?

2. Methodology

The authors employ a three-pronged approach to re-evaluate the hypothesis:

A. Theoretical Re-examination (Degrees of Freedom)

The authors challenge the theoretical argument that image embeddings have "unconstrained degrees of freedom" relative to text anchors.

• Previous Logic: If an image $I$ has a fixed distance to a text $T$, its position relative to another image $I'$ could theoretically be arbitrary.
• New Argument: In a high-dimensional space with a sufficient number of text anchors ($n_T \geq d$, where $d$ is dimensionality), the image embeddings are uniquely determined by the image-text similarity matrix ($S_{inter}$).
• Conclusion: Image-image similarities ($S_{intra}$) are not arbitrary; they are a deterministic consequence of the learned image-text structure. There are no "free" degrees of freedom allowing for arbitrary misalignment.

B. Empirical Control via Model Substitution

To isolate the effect of the training objective, the authors compare models trained only with inter-modal losses against models trained with both inter-modal and intra-modal losses.

• Models Compared:
  • Inter-modal only: CLIP, SigLIP.
  • Inter + Intra-modal: SigLIP2 (SigLIP + DINO-style loss), DINO/DINOv2 (purely intra-modal).
• Indicators Tested:
  • Cosine similarity histograms (same-class vs. different-class overlap).
  • Modality gap magnitude (distribution divergence between image-text and image-image pairs).
• Logic: If the "misalignment" indicators are specific to the lack of intra-modal training, they should disappear in SigLIP2 or DINO. If they persist, the hypothesis is flawed.

C. Alternative Method: PCA Projection

The authors propose that previous performance gains from "Modality Inversion" (converting images to text) were not due to fixing misalignment, but rather due to focusing on task-relevant axes.

• Proposal: Instead of converting images to text, project image embeddings onto the principal components of class-relevant text prompts (e.g., "a photo of [class_name]").
• Method ($PCA_{\leftarrow}$): Extract text embeddings for class names, compute the top principal components, and project image embeddings onto this subspace before measuring cosine similarity. This filters out noise and background information, isolating the dominant semantic concept.

3. Key Results

A. Refutation of Theoretical and Empirical Indicators

• Degrees of Freedom: The mathematical derivation confirms that image-image structure is fully constrained by image-text structure.
• Similarity Histograms: SigLIP (inter-modal only) and SigLIP2 (inter + intra-modal) produce identical cosine similarity distributions. Both show significant overlap between same-class and different-class pairs. This proves that high intra-class variance is a normal property of general-purpose vision encoders, not a sign of misalignment.
• Modality Gap: The divergence between image-text and image-image similarity distributions is nearly identical for both model types. The "gap" is a natural consequence of clustering in distinct manifolds, not a training defect.

B. Performance on Downstream Tasks

• Few-Shot Classification:
  • Methods designed to avoid image-image similarity (e.g., Tip-X, Coder) do not outperform simple linear classifiers (LDA) or feature selection (APE) applied directly to image embeddings.
  • SigLIP (inter-modal only) often outperforms DINOv2 (purely intra-modal) in image-image similarity tasks, contradicting the idea that inter-modal training is suboptimal for visual tasks.
• Image Retrieval:
  • The proposed $PCA_{\leftarrow}$ method consistently outperforms the state-of-the-art OTI (Optimization-based Textual Inversion) method across all datasets and models.
  • SigLIP and SigLIP2 show comparable retrieval performance, with no significant gap suggesting a "misalignment" problem in SigLIP.
  • Ablation: The $PCA_{\leftarrow}$ method improves performance only when dataset labels align with the image's dominant concept (e.g., object recognition). It fails on tasks requiring background context (e.g., weather/time classification), confirming that the gain comes from feature selection, not misalignment correction.

C. The "Dogs vs. Cats" Paradox

Re-evaluating a previous experiment where CLIP failed on a simple "Dogs vs. Cats" retrieval task:

• When tested with DINO (a strong uni-modal model), the same "failure" pattern (lower image-image retrieval scores than image-text) appeared.
• This indicates the issue is task ambiguity (images of different classes sharing semantic features) rather than a specific weakness in CLIP's alignment.

4. Key Contributions

1. Theoretical Refutation: Demonstrated mathematically that image-image similarities are uniquely determined by image-text similarities, invalidating the "unconstrained degrees of freedom" argument.
2. Empirical Evidence: Showed that "misalignment indicators" (histogram overlap, modality gap) appear equally in models with and without intra-modal training objectives, proving they are artifacts of the metrics, not the training objective.
3. Alternative Explanation: Identified task ambiguity as the true cause of performance gaps. High intra-class variance is desirable for capturing semantics beyond narrow labels.
4. Simple SOTA Method: Introduced $PCA_{\leftarrow}$, a simple projection method that achieves superior results in few-shot and retrieval tasks compared to complex text-inversion methods, without assuming any misalignment exists.

5. Significance

• Paradigm Shift: The paper challenges the prevailing belief that CLIP embeddings are "broken" for image-only tasks. It suggests that the community has been trying to "fix" a non-existent problem.
• Methodological Guidance: It argues that future work should focus on leveraging the pre-trained intra-modal geometry (e.g., via dimensionality reduction or task-specific projection) rather than circumventing it by converting images to text.
• Resource Efficiency: The proposed $PCA_{\leftarrow}$ method is computationally cheaper and easier to implement than optimization-based text inversion, yet yields better results.
• Reinterpretation of Literature: It provides a unified explanation for why "modality gap" reduction techniques often fail or even hurt performance: the gap is a feature of the representation space, not a bug.

In summary, the authors conclude that CLIP-like models are well-calibrated for intra-modal tasks, and observed performance limitations are due to the inherent ambiguity of downstream tasks and the need for better feature selection, not a fundamental misalignment in the embedding space.