🤖 machine learning

Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

This paper identifies encoder Jacobian conditioning as a critical factor in trimodal contrastive learning and proposes geometry-preserving encoders that, through simple architectural modifications, improve multimodal alignment and retrieval performance by preventing spectral collapse and amplification.

Original authors: Tillmann Rheude, Roland Eils, Benjamin Wild

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Tillmann Rheude, Roland Eils, Benjamin Wild

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world by showing it pictures, reading it stories, and listening to its heartbeat all at the same time. This is the exciting world of multimodal learning, where computers try to connect different types of information—like vision, text, and numbers—into a single, smart brain. For a while, scientists thought the secret to making these robots smarter was just inventing more complex and "expressive" ways to compare these different inputs, like creating a super-detailed scoring system to see how well a picture matches a sentence. But there's a hidden problem: even if your scoring system is perfect, the robot might still fail if the "pipes" carrying the information are clogged, broken, or leaking. In math terms, this is about how well the robot's internal pathways (called encoders) handle the flow of signals. If these pathways get twisted or blocked, the robot gets confused, no matter how good the rules are for comparing things.

This paper, titled "Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning," dives into that hidden plumbing problem. The authors, working with data from medical records and synthetic tests, discovered that the real bottleneck isn't just the complexity of the comparison rules, but the shape and stability of the robot's internal pathways. They found that when these pathways get "ill-conditioned"—a fancy way of saying they amplify some signals to infinity while crushing others to nothing—the robot's learning collapses. To fix this, they didn't invent a new, complicated scoring system. Instead, they introduced a simple architectural tweak called Geometry-Preserving Encoders (GPEs). By adding "residual paths" (which act like express lanes for information to skip over traffic jams) and using a specific type of activation function called LeakyReLU (which ensures no signal is ever completely shut off), they kept the information flow smooth and stable. The result? The robots learned much faster, understood the connections between different types of data better, and became significantly better at real-world tasks like predicting patient outcomes, proving that sometimes, keeping the pipes clean is more important than writing a fancier rulebook.

The Story of the Clogged Pipes

Let's imagine you are running a massive, high-tech library where you want to match books (text) with their movie adaptations (images) and the author's diary entries (numbers). You have a team of librarians (the encoders) whose job is to read these different things and turn them into a single "ID card" so the computer knows they belong together.

For a long time, researchers thought the key to a perfect library was to build a super-smart scoring system (the contrastive objective). They kept making these systems more complex, trying to create "expressive" rules that could catch every tiny nuance between a book and its movie. But the authors of this paper noticed something weird: even with the best scoring systems in the world, the librarians were still getting the matches wrong.

They realized the problem wasn't the scoring rules; it was the librarians themselves. Specifically, the way the librarians processed information was getting "clogged." In math language, they looked at something called the Jacobian condition number. Think of this as a measure of how much the librarian stretches or squishes the information as they pass it along.

If the condition number is good, the librarian treats all information fairly, stretching some parts a little and squishing others a little, but keeping everything recognizable.
If the condition number is bad (or "exploding"), the librarian might stretch one tiny detail into a giant mountain while squishing an entire paragraph into a speck of dust. This is called singular value collapse or explosion. When this happens, the information gets distorted so badly that the computer can't tell what it's looking at anymore.

The paper shows that in multimodal learning, where you have to juggle text, images, and numbers, these "clogged pipes" happen all the time. The standard ways of building these librarians (often using simple layers called MLPs) are surprisingly fragile. They tend to accidentally block important signals or amplify noise, leading to a breakdown in learning.

The Fix: Express Lanes and Leaky Valves

So, how did the authors fix this? They didn't try to write a better scoring rule. Instead, they redesigned the librarians' offices to keep the information flowing smoothly. They introduced Geometry-Preserving Encoders (GPEs) using two surprisingly simple tricks:

Residual Paths (The Express Lanes): Imagine a hallway where the librarian has to walk through a series of rooms to get to the exit. Sometimes, the rooms are so confusing that the librarian gets lost or tired. The authors added "express lanes" (residual connections) that let the information skip the confusing rooms and go straight to the exit, while still letting the librarian do their work on the side. This ensures that even if the complex part of the process gets messy, the original information is still there, safe and sound.
LeakyReLU (The Leaky Valves): Standard librarians use a rule called ReLU, which acts like a strict gate: if a signal is negative, it gets shut off completely (zero). The problem is, if too many signals get shut off, the whole system goes silent. The authors swapped this for LeakyReLU, which acts like a leaky valve. Even if the signal is negative, it lets a tiny bit of it through. This prevents the system from ever completely "dying" or losing track of certain directions of information.

What They Found

The authors tested these new "Geometry-Preserving" librarians on several different datasets, including:

Synthetic-XNOR: A made-up, controlled test to see how the system handles tricky logic.
MIMIC-IV & MIMIC-Symile: Real-world medical data combining things like X-rays, heartbeats, and lab reports.
UK Biobank (UKB): A massive dataset with hundreds of thousands of people, including protein data, metabolism data, and electronic health records.

The results were clear and consistent. When they used the standard, "clogged" librarians, the system struggled. The "condition numbers" (the measure of pipe health) would explode, and the accuracy would drop. But when they switched to the GPEs:

Retrieval Improved: The system got much better at finding the right matches. For example, on the UK Biobank dataset, using a method called Triangle with GPEs boosted the accuracy from about 54% to 74%.
Stability: The training process became much smoother. The "pipes" didn't clog up, and the system learned faster.
Downstream Tasks: This wasn't just about matching things; it actually made the robot smarter at predicting real outcomes. When they tested the system on predicting patient mortality (death) in a hospital, the GPEs consistently improved the results across different medical datasets.

What This Means (And What It Doesn't)

The most important takeaway from this paper is a shift in perspective. For years, the field has been obsessed with making the scoring objectives (the rules for comparison) more complex and "expressive." The authors suggest that this approach has hit a wall. They show that objective expressivity alone is insufficient. You can have the most brilliant scoring system in the world, but if the underlying "pipes" (the encoders) are broken, the system will fail.

The paper explicitly rules out the idea that we just need "bigger" or "more complex" models to solve multimodal problems. Instead, they argue that geometry preservation—keeping the internal pathways stable and well-conditioned—is the key. They demonstrate that simple architectural changes, like adding skip connections and using leaky valves, can outperform complex new scoring rules.

However, the authors are careful to note that this is a suggestion based on evidence, not a magic bullet that solves everything. They tested this on specific types of encoders (mostly MLPs and some adapted Transformers) and specific datasets (mostly medical). They don't claim that every future AI problem will be solved by this, but they strongly suggest that for multimodal learning, paying attention to the "plumbing" is just as important as designing the "rules."

In the end, this paper tells us that in the race to build smarter, multi-sensory AI, we shouldn't just focus on making the rules more complicated. Sometimes, the secret to success is just making sure the information can flow freely without getting twisted or blocked along the way.

Technical Summary: Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

Problem Statement

Multimodal contrastive learning is increasingly shifting from pairwise image-text alignment to settings involving three or more modalities (e.g., clinical reports, timeseries, and demographics in healthcare). While recent research has focused on designing increasingly expressive contrastive objectives (e.g., Symile, Triangle, Gram) to handle higher-order alignment, these models often face optimization challenges and representation degradation.

The authors identify a critical, under-explored factor: the geometric properties of the modality encoders themselves. Specifically, they posit that poorly conditioned encoder Jacobians—characterized by collapsing minimum singular values ( $\sigma_{min}$ ) and exploding maximum singular values ( $\sigma_{max}$ )—lead to ill-conditioned Jacobian condition numbers ( $\kappa$ ). This geometric degeneration impairs signal propagation, gradient transport, and ultimately, multimodal alignment, regardless of the expressivity of the contrastive objective used. This issue is particularly acute when using heterogeneous encoder families, such as Multi-Layer Perceptrons (MLPs) for tabular data, which are prone to optimization pathologies compared to architectures like Transformers.

Methodology

1. Theoretical Framework: Jacobian Conditioning

The paper defines the geometry of an encoder $E_i$ via its input-output Jacobian $J_i(x) = \frac{\partial E_i(x)}{\partial x}$ . The conditioning is quantified by the ratio of the largest to smallest singular values:
$\kappa(J_i) = \frac{\sigma_{max}(J_i)}{\sigma_{min}(J_i)}$
The authors demonstrate that in multimodal contrastive learning, gradients propagate back to inputs according to $\nabla_x L = J_i(x)^\top \nabla_{z_i} L$ . If $J_i$ is ill-conditioned (e.g., $\sigma_{min} \to 0$ due to ReLU activations zeroing out directions, or $\sigma_{max} \to \infty$ due to weight accumulation), gradient transport is degraded, leading to training instability and collapsed representations.

2. Intervention Strategies

To address this, the authors propose two complementary approaches:

Jacobian Regularization: They introduce a stochastic regularization term based on Jacobian-Vector Products (JVPs). By sampling random directions $v$ , they estimate directional gains $r(x, v) = \|J_i(x)v\|^2 / \|v\|^2$ . A loss term $L_{JVP}$ penalizes deviations from a target gain band $[m, M]$ , preventing directional collapse ( $r < m$ ) and explosion ( $r > M$ ). This serves as a mechanistic proof that conditioning influences performance.
Geometry-Preserving Encoders (GPEs): Recognizing that regularization is computationally expensive, the authors propose lightweight architectural modifications that naturally preserve Jacobian conditioning:
- Residual Transport Paths: Adding a learned projection $P_i$ around a nonlinear branch $h_i$ such that $E_i(x) = P_i x + h_i(x)$ . Theoretical bounds show that if $\|J_{h_i}\|_2 < \sigma_{min}(P_i)$ , the residual path ensures $\sigma_{min}(J_{E_i}) > 0$ , preventing total rank loss.
- LeakyReLU Activations: Replacing ReLU with LeakyReLU ( $\phi_\alpha(u)$ ) ensures that activation derivatives are never exactly zero (bounded by $\alpha$ and $1$). This prevents the activation Jacobian from completely removing input directions, a primary cause of $\sigma_{min}$ collapse in standard MLPs.

3. Experimental Setup

The authors evaluate these methods across four datasets:

Synthetic-XNOR: A controlled benchmark for higher-order alignment.
MIMIC-IV & MIMIC-Symile: Clinical benchmarks involving radiology reports, timeseries, demographics, ECGs, and chest X-rays.
UK Biobank (UKB): Large-scale population data with proteomics, metabolomics, and EHRs, including a "Union" setting with missing modalities.

They compare standard encoders against GPEs across multiple contrastive objectives (Clip, Symile, Triangle, Gram) and evaluate performance on retrieval tasks and downstream linear probes (e.g., mortality prediction).

Key Results

Training Dynamics and Jacobian Conditioning

Training dynamics analysis reveals that standard MLP encoders exhibit severe geometric degeneration:

Standard Encoders: Show rapidly increasing condition numbers ( $\kappa$ ), collapsing $\sigma_{min}$ , and exploding $\sigma_{max}$ , correlating with lower retrieval accuracy and unstable convergence.
GPEs: Maintain well-conditioned Jacobians throughout training, leading to faster convergence and higher stability.
Regularization: Direct Jacobian regularization yields similar improvements to GPEs, confirming that conditioning is a causal mechanism rather than a mere correlation.

Retrieval Performance

GPEs consistently improve retrieval performance across all datasets and objectives:

On Synthetic-XNOR, GPEs significantly boost accuracy for all objectives, with the most dramatic gains for higher-order objectives like Triangle (0.6093 $\to$ 0.9952) and Gram (0.4864 $\to$ 0.9967).
On UKB and MIMIC datasets, GPEs improve retrieval accuracy for Clip, Triangle, Gram, and Symile, even in the presence of missing modalities (UKB-Union).
Notably, while expressive objectives (like Symile) can perform well without GPEs on specific datasets (e.g., UKB), they generally underperform when paired with ill-conditioned encoders on other datasets (e.g., MIMIC-IV).

Downstream Linear Probes

A crucial finding is the differential impact of objective expressivity versus encoder geometry on downstream tasks:

Objective Expressivity: Increasingly expressive objectives primarily improve retrieval performance but yield little to no benefit in downstream linear probes (e.g., mortality prediction). Representations learned by different objectives often achieve similar linear probe performance.
Encoder Geometry: GPEs significantly improve both retrieval and downstream linear probe performance. For instance, on UKB, GPEs improved multimodal AUROC for mortality prediction across all objectives (e.g., Triangle multimodal AUROC improved from 0.7813 to 0.7969).
This suggests that preserving encoder geometry is essential for learning representations that generalize to downstream tasks, whereas objective expressivity alone is insufficient.

Significance and Claims

The paper claims that the field of multimodal contrastive learning has over-emphasized the design of expressive scoring functions while neglecting the geometric and optimization properties of the underlying encoders.

Mechanistic Insight: The authors establish that encoder Jacobian conditioning is a key factor governing the trainability of multimodal contrastive learning, particularly for MLP-based encoders common in processing tabular and structured data.
Practical Solution: They demonstrate that simple, computationally efficient architectural changes (residual paths and LeakyReLU) can recover the geometric benefits of complex regularization, making them a practical intervention for improving multimodal models.
Broader Implication: The results suggest a paradigm shift: successful multimodal learning depends not just on what is aligned (the objective), but how it is aligned (the encoder geometry). The authors argue that future progress may rely more on understanding information transport and optimization stability than on designing increasingly complex contrastive objectives.

The authors remain modest, noting that their analysis focuses primarily on MLP-based projections and healthcare datasets, and that Transformer-based encoders may already possess inherent geometry-preserving mechanisms (via skip connections) that make them more robust to these issues.