📊 statistics

Expressivity In Multimodal Contrastive Learning

This paper analyzes the representational capacity of multimodal contrastive learning, demonstrating that while standard CLIP is a universal approximator for two modalities, its common multi-modal extension fails to capture arbitrary joint distributions, leading the authors to propose Hadamard-CLIP as a simple modification that restores universal approximation for any number of modalities without sacrificing retrieval efficiency.

Original authors: Andrew Stuart, Florian Wolf

Published 2026-08-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andrew Stuart, Florian Wolf

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of artificial intelligence, machines are increasingly learning to see, hear, and read simultaneously. Instead of just recognizing a picture or translating a sentence in isolation, these systems are being trained to understand how different types of information connect. A computer might look at an image of a sunset and learn to link it with the words "golden hour," or associate a specific sound with a visual scene. This process, known as contrastive learning, has become a cornerstone of how computers build a shared understanding of the world. It works by teaching the machine to pull related pieces of information closer together in its internal memory while pushing unrelated ones apart. The result is a powerful ability to retrieve images using text, generate pictures from descriptions, or navigate complex data landscapes. However, while these systems have achieved remarkable success in practice, scientists have long wondered about the fundamental limits of their understanding. Specifically, they wanted to know if the mathematical structures used to build these systems were actually capable of capturing every possible relationship between different types of data, or if there were hidden blind spots in their design.

A team of researchers at the California Institute of Technology set out to answer this question by looking at the architecture of these learning systems not as a collection of code, but as a way of estimating probabilities. They imagined a scenario where the machine has seen an infinite amount of data, allowing them to strip away the noise of limited samples and focus purely on the system's theoretical capacity. Their investigation revealed a sharp divide in how these machines handle information depending on the number of data types involved. When the system deals with just two types of data, such as images and text, the standard architecture used today is theoretically perfect. It is powerful enough to learn any possible relationship between the two, matching the capability of the most complex, general-purpose networks. This finding provides a solid mathematical foundation for the success of current two-part systems.

The story changes, however, when the researchers examined systems designed to handle three or more types of data at once, such as combining video, audio, and text. The most common approach used in practice for these complex tasks involves a method that simply adds up the similarities between every possible pair of data types. The researchers proved that while this method is excellent at learning how any two specific types of data relate to each other, it is fundamentally incapable of understanding the full picture when all three or more are present together. It creates a blind spot where the system can understand the relationship between A and B, and between B and C, but fails to grasp the unique, three-way connection that exists only when A, B, and C are considered simultaneously. The standard method is mathematically proven to miss these higher-order interactions, no matter how much data it is fed or how long it is trained.

To solve this problem, the researchers proposed a simple but powerful modification to the existing design. They introduced a new architecture that adds a single, learned set of weights to the top of the standard system. This small addition acts as a bridge, allowing the machine to combine the information from all data types in a way that captures their full, complex relationship. The new design, which they named Hadamard-CLIP, restores the system's ability to learn any possible joint distribution of data, regardless of how many types are involved. Crucially, this fix does not sacrifice the speed that makes these systems useful in the real world. Unlike other complex solutions that would require the machine to process raw data every time it makes a prediction, this new method allows the system to pre-compute and store simplified representations of the data. This means that even with the added complexity of understanding multiple modalities, the system can still retrieve information as quickly as the original, simpler models. The work demonstrates that by making a minimal structural change, it is possible to unlock the full expressive power of multimodal learning, ensuring that machines can truly understand the intricate web of connections between the many different ways we describe the world.

Technical Summary: Expressivity In Multimodal Contrastive Learning

Problem Statement

Contrastive learning, particularly the CLIP architecture, has become a foundational technique for learning joint representations across modalities (e.g., text and images). While empirically successful, the theoretical limits of its expressivity—specifically, its capacity to approximate arbitrary joint probability distributions of multiple modalities—remain poorly understood.

Existing literature often focuses on information-theoretic bounds (e.g., mutual information maximization) or downstream task performance. However, these analyses often presuppose that the chosen parameterization can represent the target relationship. This paper addresses a foundational gap: Is the contrastive parameterization itself expressive enough to universally approximate the underlying joint distribution, or is it limited to approximating only certain conditional distributions?

The authors adopt a population-level, density-estimation viewpoint. Instead of analyzing finite-sample estimation or optimization dynamics, they treat the data as a known joint probability measure $\mu$ and ask whether a given contrastive architecture can approximate $\mu$ arbitrarily well by learning a "tilting function" $\Phi$ that adjusts the product of marginals $\mu_P$ to match the joint $\mu$ . The measure of expressivity is the infimum of the Kullback-Leibler (KL) divergence $D_{KL}(\mu \parallel \mu_a)$ over the parameter space.

Methodology

The paper frames contrastive learning as learning a tilting function $\Phi$ such that the approximate measure is defined as:
$\frac{d\mu_a}{d\mu_P}(x) = \frac{1}{Z} \exp(\Phi_a(x))$
where $\Phi$ is the log-density ratio (tilting function) of the true joint relative to the product of marginals.

The authors establish a hierarchy of theoretical results:

Loss Hierarchy: They prove that minimizing the joint KL divergence $D_{KL}(\mu \parallel \mu_a)$ strictly upper-bounds the conditional losses (both full-conditionals and pairwise-conditionals) used in practical contrastive learning. Thus, if an architecture cannot approximate the joint, it cannot minimize the joint KL divergence to zero, even if it minimizes conditional losses.
Function-to-Measure Approximation: They demonstrate that if a parametric family of functions can uniformly approximate continuous tilting functions (Universal Approximation in $C_0$ ), then the induced family of measures can approximate the true joint distribution in KL divergence.
Architecture Analysis: They apply these general theorems to specific architectures:
- Bimodal CLIP ( $m=2$ ): The standard two-tower architecture with inner-product scoring.
- Sum-of-Pairs CLIP ( $m \ge 3$ ): The common extension where the score is the sum of all pairwise inner products.
- Hadamard-CLIP (H-CLIP): A proposed modification adding a learned weight vector to the element-wise (Hadamard) product of encoder outputs.

Key Contributions and Results

1. Bimodal CLIP is a Universal Approximator

For two modalities ( $m=2$ ), the paper proves that the standard two-tower CLIP architecture is as expressive as any general joint network.

Result: The family of functions defined by $\Phi_{2\text{-clip}}(x_1, x_2) = \langle \bar{g}_1(x_1), \bar{g}_2(x_2) \rangle / \tau$ is dense in the space of continuous functions.
Implication: Provided sufficient capacity and a learnable temperature $\tau$ , the bimodal CLIP can approximate any joint distribution arbitrarily well. This provides a theoretical justification for the empirical success of CLIP in text-image tasks.
Caveat: The authors note that this does not guarantee dimensionality reduction; in some cases, the latent dimension required for perfect approximation may equal or exceed the dimension of the original joint domain.

2. Sum-of-Pairs CLIP Fails for $m \ge 3$

The paper demonstrates that the widely used generalization of CLIP for three or more modalities, which sums pairwise similarities ( $\Phi_{m\text{-clip}} = \sum_{i<j} \langle \bar{g}_i, \bar{g}_j \rangle$ ), cannot universally approximate arbitrary joint distributions.

Result: While this architecture can approximate all pairwise conditionals ( $x_i | x_j$ ) arbitrarily well, it structurally fails to capture higher-order interactions (e.g., three-way interactions like $x_1 x_2 x_3$ ). Consequently, it cannot approximate the full joint distribution or the full conditional ( $x_i | x_{-i}$ ).
Significance: This reveals a theoretical limitation in current multimodal practices: the sum-of-pairs loss is insufficient for modeling complex dependencies among three or more modalities, despite its empirical utility.

3. Hadamard-CLIP (H-CLIP) Restores Universal Approximation

To address the failure of the sum-of-pairs approach, the authors propose Hadamard-CLIP.

Architecture: This method adds a single learned weight vector $\omega \in \mathbb{R}^{d_{Lat}}$ to the element-wise product of all encoder outputs:
$\Phi_{Had}(x_1, \dots, x_m) = \omega^\top (\bar{g}_1(x_1) \odot \dots \odot \bar{g}_m(x_m))$
Result: The authors prove that H-CLIP restores universal approximation for any number of modalities $m$ . It can approximate the full joint distribution arbitrarily well.
Efficiency: Crucially, H-CLIP preserves the inference-time efficiency of standard CLIP. Unlike a general joint network (which requires processing all raw modalities simultaneously), H-CLIP allows for pre-computable embeddings. One can pre-compute the Hadamard product of all modalities except the query modality, enabling fast retrieval via a single dot product, similar to standard CLIP.

Significance and Claims

The paper claims to provide a foundational theoretical understanding of the representational capacity of contrastive learning architectures, isolating expressivity from finite-sample estimation and optimization issues.

Theoretical Gap: It identifies a specific structural failure in the standard multimodal extension of CLIP (sum-of-pairs) that prevents it from modeling arbitrary joint distributions, a gap previously uncharacterized in the literature.
Minimal Modification: It proposes H-CLIP as a minimal, theoretically sound modification that restores universal approximation without sacrificing the computational efficiency (pre-computable embeddings) that makes contrastive learning scalable.
Scope: The authors explicitly state that their results are limited to the population limit (infinite data). They do not claim to solve issues related to finite-sample generalization, optimization convergence, or specific downstream task performance guarantees, though they suggest these are natural avenues for future work.

In summary, the paper argues that while bimodal CLIP is theoretically sufficient for universal approximation, current multimodal extensions are not, and a simple tensor-decomposition-inspired fusion (H-CLIP) is required to achieve full expressivity while maintaining practical efficiency.