🤖 machine learning

A Unified Geometric Framework for Weighted Contrastive Learning

This paper establishes a unified geometric framework interpreting weighted contrastive learning as Distance Geometry Problems, revealing how specific weighting schemes determine whether optimal embeddings achieve ideal geometries like regular simplices or suffer from degeneracy and inconsistency due to class imbalance or label mismatch.

Original authors: Raphael Vock, Edouard Duchesnay, Benoit Dufumier

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Raphael Vock, Edouard Duchesnay, Benoit Dufumier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an architect trying to build a city (the "embedding space") where every building represents a piece of data, like a photo of a cat or a sentence about a dog. Your goal is to arrange these buildings so that similar things are close together and different things are far apart.

This paper, titled "A Unified Geometric Framework for Weighted Contrastive Learning," acts like a master blueprint for that city. It argues that the rules you use to decide which buildings should be neighbors (the "weighting scheme") dictate exactly what the final city will look like. Sometimes, the rules work perfectly; other times, they force the city into a shape that simply cannot exist, leading to a collapsed or distorted mess.

Here is the breakdown of their findings using everyday analogies:

1. The Core Idea: The Blueprint vs. The Reality

In the world of AI, "Contrastive Learning" is the process of teaching a computer to understand relationships. The authors say this is actually a Distance Geometry Problem.

The Analogy: Imagine you have a list of instructions saying, "The library must be 5 miles from the school," and "The park must be 2 miles from the library." This is your Weighting Scheme (the blueprint).
The Goal: You want to draw a map (the Embedding) where these distances are real.
The Problem: The paper shows that depending on how you write those instructions, you might be asking for something impossible. For example, if you say "A is 1 mile from B, B is 1 mile from C, but A is 10 miles from C," you can't draw that on a flat map without breaking the rules of geometry.

2. The "SupCon" vs. "Soft SupCon" Experiment (The Class Imbalance Trap)

The paper looks at how AI handles different groups of data, like sorting images of cats, dogs, and birds.

The Scenario: Imagine you have a classroom with 100 students. 90 are in the "Cat" group, and only 1 is in the "Dog" group.
The Old Way (SupCon): The standard method treats the "Cat" group as a single tight cluster and the "Dog" as another. However, because there are so many cats, the "Cat" cluster gets squished together, and the distance between the "Cat" cluster and the "Dog" cluster gets distorted. It's like trying to fit a huge crowd and one lonely person into a dance circle; the crowd pushes the lonely person into a weird spot. The paper proves this creates a distorted geometry where the "Dog" isn't just far away; it's in the wrong place relative to the size of the crowd.
The New Way (Soft SupCon): The authors propose a "soft" version. Instead of saying "Cats are 100% similar and Dogs are 0% similar," they say "Cats are 100% similar, but Dogs are almost 0% similar (maybe 0.1%)."
The Result: This tiny bit of "softness" acts like a shock absorber. Even with the imbalanced class sizes, the city layout remains a perfect, symmetrical shape (a Regular Simplex). It's like giving the lonely dog a tiny bit of space to breathe so the whole dance circle stays balanced.

3. The "y-Aware" Mistake (The Sphere vs. The Flat Map)

The paper also looks at cases where the data isn't just categories (Cat/Dog) but continuous values, like the "thickness" of a line in a drawing.

The Mismatch: Some methods try to measure similarity in the AI's brain using Cosine Similarity (which assumes everything lives on the surface of a sphere, like points on a globe). However, they use Euclidean Distance for the labels (which assumes a flat map, like a piece of graph paper).
The Analogy: This is like trying to measure the distance between New York and London using a flat ruler on a piece of paper, but then forcing the result to fit onto a globe. The paper proves that unless your data points happen to already lie perfectly on a sphere, you cannot reach the perfect solution. The AI is trying to solve a puzzle where the pieces don't fit the box.
The Fix: The authors show that if you match the tools to the job—using "Flat Map" math for "Flat Map" data, or "Globe" math for "Globe" data—the AI can find the perfect, unique solution.

4. How to Measure Success

Finally, the paper introduces three new "rulers" to check if the AI has built the city correctly.

Instead of just asking, "Did the AI get the right answer?" they ask, "Does the shape of the AI's internal map match the shape of the blueprint?"
They use metrics like Procrustes Similarity (which is like taking a photo of the AI's map, rotating and resizing it, and seeing if it perfectly overlays the ideal map).

Summary

The paper's main takeaway is simple: The rules you give the AI determine the shape of its world.

If your rules are "hard" and ignore imbalances, the world gets distorted.
If your rules mix "flat" and "spherical" math, the world breaks.
If your rules are "soft" and geometrically consistent, the AI builds a perfect, stable city where every piece of data sits exactly where it should.

The authors aren't just saying "this works better"; they are providing the mathematical proof of why certain setups collapse and others succeed, giving engineers a principled way to design better AI systems.

Technical Summary: A Unified Geometric Framework for Weighted Contrastive Learning

Problem Statement

Contrastive Learning (CL) has become a cornerstone of supervised and self-supervised representation learning, yet the geometric structure of the embeddings it produces remains poorly understood. While empirical successes exist (e.g., SimCLR, SupCon, CLIP), it is unclear what specific relational structures these objectives implicitly attempt to realize in the embedding space, whether such structures are always achievable, and how weighting schemes influence the geometry of the solution. Specifically, there is a lack of rigorous characterization regarding the existence, uniqueness, and distance structure of optimal representations under various weighting schemes, particularly in the presence of class imbalance or continuous labels.

Methodology

The authors propose interpreting weighted Contrastive Learning as a Distance Geometry Problem (DGP). By analyzing the weighted InfoNCE (w-InfoNCE) loss, they establish that minimizing the loss is equivalent to finding a geometric realization of a target pairwise geometry defined by the weighting matrix $W$ .

Core Framework

Unified Formulation: The paper defines the w-InfoNCE loss using a similarity matrix $W$ (encoding prior knowledge via labels or metadata) and a latent similarity matrix $S$ .
Entropic Lower Bound: Under the assumption of strictly positive weights, the authors derive an entropic lower bound for the loss. They prove that the global minimum is attained when the latent similarities $s_{ij}$ satisfy $s^*_{ij} = \log(w_{ij}) + c$ .
Connection to DGP: This condition transforms the optimization problem into a DGP. Depending on the similarity metric used (Euclidean distance vs. Cosine similarity), the problem reduces to finding an embedding that satisfies prescribed Euclidean distances or cosine similarities, respectively.
Existence and Uniqueness: The authors apply computational geometry concepts (Euclidean Distance Matrices and Spherical EDMs) to determine conditions under which optimal embeddings exist and are unique. They distinguish between "realizable" geometries (where a solution exists), "degenerate" solutions (where multiple qualitatively different configurations minimize the loss), and "inconsistent" objectives (where the target geometry cannot be realized in the embedding space).

Key Contributions

1. Geometric Characterization of Supervised Classification

The paper analyzes SupCon (hard weights) and Soft SupCon (dense weights with small non-zero similarity for negative pairs):

Class Collapse: Both objectives cause samples within the same class to collapse to a single prototype.
Class Imbalance Effects:
- SupCon: In the presence of class imbalance, SupCon induces non-uniform inter-class similarities. The geometry of the class prototypes depends on class sizes, breaking the symmetry of a regular simplex. Minority classes are grouped more closely.
- Soft SupCon: By assigning a small non-zero similarity to inter-class pairs, Soft SupCon preserves a regular simplex geometry regardless of class imbalance, provided the temperature parameter is tuned correctly.

2. Analysis of Continuous-Label Settings

The authors examine objectives for regression and weakly supervised learning:

y-Aware CL: This method uses Euclidean distances in the label space ( $W$ ) but cosine similarity in the latent space ( $S$ ). The paper proves this combination is geometrically inconsistent. Unless the labels lie on a hypersphere, the objective cannot attain its entropic lower bound because Euclidean label distances cannot be perfectly mapped to spherical latent similarities.
Geometrically Consistent Alternatives:
- Euclidean–Euclidean: Using Euclidean distance in both spaces allows for a unique, realizable optimum.
- X-CLR: Using cosine similarity in both label and latent spaces (spherical–spherical) admits a unique optimal embedding, provided the label dimension and temperature constraints are met.

3. New Evaluation Metrics

To empirically validate these theoretical claims, the authors introduce three geometric metrics:

Entropic Loss Gap: Measures how close the achieved loss is to the theoretical lower bound.
Similarity Coefficient ( $r^2_{s-sim}$ ): Measures the correlation between pairwise similarities in the learned embeddings versus the theoretical optimum.
Procrustes Similarity ( $r^2_{Proc}$ ): Measures the alignment between the learned embeddings and the theoretical optimum after optimal rigid transformation (rotation/translation).

Experimental Results

The authors validate their framework on the MNIST dataset and CIFAR-100:

Discrete Classification: In balanced settings, learned embeddings converge to a regular simplex. In imbalanced settings, SupCon embeddings distort according to class sizes, while Soft SupCon maintains the regular simplex structure.
Continuous Regression: When using Euclidean weights with Euclidean latent similarity, the learned embeddings closely approximate the label geometry ( $r^2_{Proc} \approx 0.96$ ). Conversely, using the inconsistent y-Aware setup (Euclidean labels, spherical latent) fails to recover the correct geometry, whereas the corrected Euclidean-Euclidean setup succeeds.
X-CLR: Experiments on CIFAR-100 show that models trained with consistent label-embedding geometries (e.g., text embeddings matching the latent space geometry) achieve better generalization and higher Procrustes similarity scores compared to mismatched geometries.

Significance and Claims

The paper claims to provide a principled framework for designing contrastive objectives by explicitly matching the geometry induced by the weighting scheme with the geometry of the embedding space.

Realizability: The choice of weighting scheme determines whether a contrastive objective is geometrically realizable, degenerate, or inconsistent.
Design Guidance: The authors argue that practitioners should choose similarity functions for $W$ and $S$ that are consistent (e.g., Euclidean-Euclidean or Spherical-Spherical) to ensure the existence of a unique, optimal solution.
Theoretical Insight: This work offers a complementary perspective to existing information-theoretic and probabilistic views of CL, focusing on the structural properties of the solution space. It explains why certain objectives (like Soft SupCon or X-CLR) may outperform others (like standard SupCon in imbalanced settings or y-Aware with mismatched metrics) by virtue of their geometric realizability.

The authors acknowledge limitations, noting that their analysis assumes explicit dissimilarity matrices derived from labels or metadata, whereas self-supervised settings (like SimCLR) rely on implicit, data-driven dissimilarity matrices that are harder to characterize theoretically. They position this work as a foundation for future research into the geometric properties of implicit contrastive objectives.