Technical Summary: The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

Problem Statement

Contrastive learning (CL) has emerged as a dominant paradigm for self-supervised representation learning, yet the theoretical conditions under which it successfully recovers meaningful latent geometry remain incompletely understood. While empirical success is widespread, the precise mechanisms driving CL are often heuristic, leading to inefficient resource use and design choices that may not fully exploit the method's potential.

Existing theoretical explanations generally fall into two categories:

1. The Invariance Explanation: Posits that CL induces representations invariant to nuisance factors. However, this framework does not specify which factors are nuisance, nor how data augmentations implicitly determine this partition. Furthermore, factors deemed "nuisance" by the contrastive objective may carry discriminative information necessary for specific downstream tasks.
2. The Recovery Explanation: Assumes data lies on a high-dimensional manifold with a low-dimensional latent structure, and that CL recovers this latent structure. While promising, prior identifiability results (e.g., Zimmermann et al., 2021) rely on strong assumptions, specifically that positive-pair sampling follows a full-support von Mises-Fisher (vMF) distribution.

The core problem addressed is that real-world sampling mechanisms (augmentations) rarely satisfy the full-support assumption. They often restrict the support of positive pairs (e.g., by fixing certain semantic features while perturbing others). The paper investigates whether the contrastive objective alone is sufficient to recover the latent geometry when this "diversity condition" is violated, and how architectural inductive bias interacts with sampling diversity.

Methodology

Theoretical Framework

The authors develop a measure-theoretic framework to analyze contrastive learning, modeling the data generation process as an injective mapping $g: \mathcal{Z} \to \mathcal{X}$ from a latent space $\mathcal{Z}$ to an observation space $\mathcal{X}$. The encoder $f: \mathcal{X} \to \mathcal{Z}$ aims to learn a recovery map $h = f \circ g$.

Key Definitions:

• Diversity Condition: A support requirement on the conditional law of positive pairs $P_{\tilde{Z}|Z}$. It holds if, for almost every anchor $z$, the marginal measure $P_Z$ is absolutely continuous with respect to the conditional measure $P_{\tilde{Z}|z}$ ($P_Z \ll P_{\tilde{Z}|z}$). Intuitively, the view-generation process must perturb all latent features; if any latent dimension remains fixed, the encoder cannot distinguish variations along that dimension.
• InfoNCE Loss: Analyzed as a cross-entropy minimization problem between the true sampling conditional $P_{\tilde{Z}|z}$ and the model conditional $Q_{h,z}$.

Theoretical Analysis

The paper proceeds through three analytical stages:

1. Full-Support Case: It is proven that the standard full-support vMF setting implies the diversity condition. Using a probabilistic Mazur-Ulam argument, the authors show that any global minimizer of the asymptotic contrastive loss recovers the latent space up to an orthogonal transformation (isometry).
2. Violated Diversity Case: The authors model scenarios where the conditional support is restricted to a lower-dimensional submanifold (e.g., fixing latent coordinates $u$ while varying $v$). They prove that under violated diversity, non-orthogonal maps can attain strictly lower asymptotic contrastive loss than any orthogonal map. Specifically, the loss function actively disincentivizes geometry-preserving solutions, leading to poorly structured latent spaces where semantic relationships are not preserved.
3. Corrected Model: To address the support mismatch, the authors introduce a "support-corrected" InfoNCE variant where negative samples are drawn from the same constrained submanifold as the positive pairs. They prove that this correction makes isometric (orthogonal) solutions achievable (i.e., they become global minimizers), but crucially, it does not guarantee that they are the unique minimizers.

Experimental Validation

The theoretical predictions are validated through two sets of experiments:

1. Synthetic Benchmarks: Using a spherical latent space ($S^2$) and five generative processes (Identity, Linear, Spiral, Patches, Invertible MLP), the authors compare low-inductive-bias encoders (MLP) against high-inductive-bias encoders (inverse models designed to mirror the generative process).
   • Metrics: Linear identifiability ($R^2$), Mean Correlation Coefficient (MCC), and Angular Preservation Error (APE).
   • Conditions: Sampling with diversity held (full support) vs. diversity violated (fixed dimensions).
2. CIFAR-10 Experiments: Using SimCLR with three architectures (MLP, ViT, ResNet-18) and varying augmentation regimes (All, Crop Only, All w/o crop).
   • Goal: To qualitatively examine how architectural inductive bias compensates for limited sampling diversity in real-world data where ground-truth latents are unknown.

Key Results

Theoretical Findings

• Necessity of Diversity: The diversity condition is necessary for isometric latent recovery. When violated, the contrastive objective alone fails to recover the latent geometry, even with infinite data and expressive encoders.
• Failure of Standard InfoNCE: Under violated diversity, the standard InfoNCE loss favors geometry-distorting maps over orthogonal ones.
• Role of Support Correction: A support-corrected InfoNCE objective (sampling negatives from the constrained manifold) restores the achievability of isometric solutions but does not ensure their selection. The solution space remains non-unique without additional constraints.
• Compensatory Role of Inductive Bias: When sampling diversity is insufficient, architectural inductive bias becomes critical. High-inductive-bias encoders (those approximating the inverse of the generative process) can recover the latent structure even when the diversity condition is violated, whereas low-inductive-bias encoders fail.

Empirical Findings

• Synthetic Results:
  • Diversity Holds: MLP encoders achieve near-perfect linear identifiability ($R^2 \ge 0.99$), confirming that expressive encoders can recover latent geometry when the diversity condition is met.
  • Diversity Violated: MLP performance collapses ($R^2 \in [0.05, 0.13]$ for most processes). In contrast, inverse encoders (high inductive bias) maintain high recovery performance ($R^2 \ge 0.88$).
  • Adapted InfoNCE: Applying the support-corrected loss partially recovers MLP performance ($R^2 \approx 0.60\text{--}0.65$) but does not reach the performance of high-inductive-bias encoders, confirming that loss correction alone is insufficient for unique selection.
• CIFAR-10 Results:
  • Broader sampling support ("All" augmentations) yields the highest downstream accuracy across all architectures.
  • Higher inductive bias (ResNet-18 > ViT > MLP) consistently improves performance.
  • Interaction Effect: The performance gap between architectures widens as the diversity condition is increasingly violated (e.g., "All w/o crop"). Under restricted sampling, the high-inductive-bias ResNet-18 significantly outperforms the low-inductive-bias MLP, demonstrating that architectural priors are critical when sampling diversity is limited.

Significance and Claims

The paper claims to provide a rigorous framework for understanding the regimes of success and failure in contrastive learning. Its primary contributions are:

1. Formalizing the Diversity Condition: Distinguishing it from the full-support vMF assumption used in prior work and demonstrating that its violation fundamentally alters the optimization landscape, making geometry-preserving solutions suboptimal under standard objectives.
2. Clarifying the Role of Inductive Bias: Establishing that when sampling diversity is limited, architectural inductive bias is not merely a heuristic improvement but a necessary mechanism to recover meaningful latent structures.
3. Limitations of Loss Modification: Showing that while correcting the support mismatch in the loss function makes isometric solutions achievable, it does not guarantee their selection, reinforcing the necessity of inductive bias.

The authors conclude that the interplay between sampling mechanisms and encoder inductive bias jointly determines representation quality. They suggest that design choices for augmentation pipelines and encoder architectures should be guided by this theoretical alignment, potentially reducing computational waste from trial-and-error experimentation. The work does not claim to solve all identifiability issues but rather clarifies the specific conditions under which contrastive learning succeeds or fails to recover latent geometry.