🤖 machine learning

Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse

This paper theoretically proves that hard-negative sampling in contrastive learning drives representations toward optimal Neural-Collapse geometry while preventing Dimensional-Collapse, a phenomenon empirically validated through Adam optimization with feature normalization.

Original authors: Ruijie Jiang, Thuan Nguyen, Shuchin Aeron, Prakash Ishwar

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruijie Jiang, Thuan Nguyen, Shuchin Aeron, Prakash Ishwar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to organize a massive library of books. Your goal is to create a system where books on the same topic (like "Cooking") are stacked neatly together, while books on different topics (like "Cooking" vs. "Space Travel") are pushed as far apart as possible on the shelves.

This is essentially what Contrastive Learning does for computers. It teaches a computer to understand data by learning which pieces are similar (positives) and which are different (negatives).

This paper investigates a specific, tricky strategy called Hard-Negative Sampling. Let's break down what the authors found using simple analogies.

1. The Two Types of "Negatives"

In this library analogy, a "negative" is a book you pick to show the computer: "This is not a cooking book."

Easy Negatives: You pick a book about "Space Travel." It's obviously not cooking. The computer learns this easily.
Hard Negatives: You pick a book about "Baking" (which is cooking) but label it as "Not Cooking" to trick the computer, or you pick a book about "Food History" that looks very similar to a "Recipe" book. These are "hard" because they are confusing.

The paper asks: Is it better to use the confusing "Hard Negatives" to train the computer?

2. The "Perfect Arrangement" (Neural Collapse)

The authors discovered a "Gold Standard" for how the computer should organize these books. They call this Neural Collapse.

Imagine the perfect library:

All "Cooking" books are stacked into one single, tiny, perfect pile.
All "Space" books are stacked into another tiny pile.
These piles are arranged in a perfect geometric shape (like the points of a star or a soccer ball) where every pile is exactly the same distance from every other pile.

The paper proves mathematically that if you want the computer to learn the absolute best way to organize data, it must end up in this "Neural Collapse" state.

3. The Big Surprise: Hard Negatives are a Double-Edged Sword

The authors ran experiments to see if using "Hard Negatives" helps the computer reach this perfect "Neural Collapse" state.

The Good News (Supervised Learning):
When the computer has a teacher (labels) telling it exactly what category a book belongs to, using Hard Negatives works like a charm. It pushes the computer toward that perfect geometric arrangement. It's like a strict coach who forces the students to study the hardest exam questions; they learn the material perfectly.

The Bad News (Unsupervised Learning):
When the computer has no teacher (no labels) and has to guess the categories itself, Hard Negatives can be a disaster.

Without a teacher, the computer gets confused by the "Hard Negatives."
Instead of spreading the books out into a perfect star shape, the books collapse into a messy, flat pile.
The authors call this Dimensional Collapse. Imagine trying to arrange a 3D sculpture, but the computer accidentally squashes it flat onto a 2D piece of paper. It loses all its depth and structure.

4. The Magic Ingredient: Normalization

The paper found a simple fix for the "squashed" problem. It's called Feature Normalization.

Think of this as forcing every book to be exactly the same size and weight before you put it on the shelf.

With Normalization: Even when using Hard Negatives without a teacher, the computer can still find the perfect arrangement. The "books" stay organized.
Without Normalization: The computer fails. The "Hard Negatives" cause the books to collapse into a flat, useless mess.

5. The "Tightrope" of Difficulty

The paper also found that the "Hardness" of the negative samples needs to be just right.

If the negatives are too easy, the computer doesn't learn much.
If they are too hard (especially without a teacher), the computer gets confused and collapses.
There is a "sweet spot" (a moderate level of difficulty) where the computer learns the best.

Summary of the Paper's Claims

The Goal: The best way for a computer to organize data is a specific, perfect geometric shape called Neural Collapse.
The Theory: The authors proved mathematically that Hard-Negative Sampling should lead to this perfect shape, but only if the math works out perfectly.
The Reality Check: In real-world experiments:
- With a Teacher (Labels): Hard Negatives help the computer reach the perfect shape.
- Without a Teacher (No Labels): Hard Negatives often cause the computer to "collapse" into a flat, broken shape, unless you use a specific trick called Normalization.
The Conclusion: Hard-Negative Sampling is a powerful tool, but it's dangerous if you don't use it correctly (specifically, without normalization in unsupervised settings). It's like using a sledgehammer to fix a watch; it works great if you know exactly what you're doing, but it can easily break the machine if you don't.

The paper concludes by saying that while we know when this perfect arrangement happens, figuring out exactly how to guarantee it in every messy real-world situation is still a puzzle for future researchers.

Technical Summary: Hard-Negative Sampling for Contrastive Learning

Problem Statement

Contrastive Learning (CL) aims to learn representations where similar data points are close and dissimilar ones are far apart. While hard-negative sampling (selecting negative samples that are difficult to distinguish from the anchor) is empirically known to improve downstream performance, its theoretical impact on the geometry of global minima remains under-explored. Specifically, there is a lack of rigorous understanding regarding:

Whether hard-negative sampling alters the optimal representation geometry compared to standard CL.
The conditions under which Neural Collapse (NC)—a geometry where class means form an Equiangular Tight Frame (ETF) and intra-class variance vanishes—occurs in both Supervised (SCL) and Unsupervised (UCL) settings.
The phenomenon of Dimensional Collapse (DC), where representations collapse to a lower-dimensional subspace, and how hard-negative sampling and feature normalization influence this.

Existing theoretical works often rely on strong assumptions, such as the conditional independence of augmented positive views given the label, and focus on specific loss functions like InfoNCE.

Methodology

The authors propose a unified theoretical framework to analyze CL losses under a general latent data model.

1. Generalized Loss and Hardening Functions

The paper defines a general contrastive loss function $\ell_k$ using a convex, argument-wise non-decreasing function $\psi_k$ . This subsumes popular losses like InfoNCE and Triplet Loss.
To model hard-negative sampling, the authors introduce a hardening function $\eta: \mathbb{R}^k \to \mathbb{R}$ , which is non-negative and argument-wise non-decreasing. This function re-weights the sampling distribution of negative samples, increasing the probability of selecting negatives that are more aligned with the anchor in the representation space.

2. Theoretical Framework

The analysis covers four settings:

SCL: Supervised Contrastive Learning (labels available).
HSCL: Hard-negative SCL.
UCL: Unsupervised Contrastive Learning (latent labels).
HUCL: Hard-negative UCL.

The authors derive lower bounds for the expected losses in these settings. A key theoretical tool is the Harris comparison inequality, used to prove that the expected loss with hard-negative sampling is lower-bounded by the standard CL loss.

3. Optimization and Geometry

The paper investigates the global minima of these losses. It defines Neural Collapse (NC) as the state where:

Class means form a normalized Equiangular Tight Frame (ETF).
Within-class variance is zero (all samples of a class map to the same point).
The representation dimension $d_Z \ge C-1$ (where $C$ is the number of classes).

The authors also analyze Dimensional Collapse (DC), defined as the empirical covariance matrix of class means having singular values that are zero or orders of magnitude smaller than the largest, indicating a loss of representational capacity.

Key Contributions

1. Theoretical Lower Bounds and Dominance (Theorem 1)

The authors prove that for any representation mapping $f$ , the losses of Hard-SCL (HSCL) and Hard-UCL (HUCL) are lower-bounded by their standard counterparts (SCL and UCL). This is established using the Harris inequality, demonstrating that hard-negative sampling effectively "tilts" the sampling distribution to minimize the loss further, provided the loss function is argument-wise non-decreasing.

2. Optimality of Neural Collapse (Theorems 2 & 3)

SCL/HSCL: The paper proves that the global minima for both SCL and HSCL correspond to the Neural Collapse geometry. Crucially, this result does not require the assumption that positive samples are conditionally independent given the label, a common constraint in prior literature. The optimal loss value is identical for both SCL and HSCL when NC is achieved.
UCL/HUCL: The authors establish that NC is the optimal geometry for UCL if the embedding dimension is sufficiently large relative to the number of latent classes. For HUCL, the optimal geometry depends on the hardness level and the possibility of "class collision" (negative samples sharing the latent class with the anchor).

3. Empirical Validation of Optimization Dynamics

The paper provides the first empirical demonstration that:

Adam Optimization with Hard Negatives: Using Adam with random initialization and suitable hardness levels, HSCL and HUCL can converge to the NC geometry, provided feature normalization (unit-ball or unit-sphere) is applied.
Dimensional Collapse: Without hard-negative sampling or feature normalization, representations learned via Adam suffer from Dimensional Collapse (DC), failing to attain the NC geometry.
Normalization: Feature normalization is critical for mitigating DC. Without it, both random and hard-negative methods fail to achieve NC.

Experimental Results

Synthetic Data

SCL/HSCL: Converged to NC regardless of the hardness level ( $\beta$ ).
UCL: Converged to NC only when positive pairs were constructed using label information (satisfying theoretical assumptions). When using additive Gaussian noise (violating conditional independence), NC was not observed.
HUCL: Did not converge to NC; class means deviated significantly from ETF geometry, with deviation increasing as hardness increased.

Real-World Datasets (CIFAR100, CIFAR10, TinyImageNet)

Downstream Performance: Hard-negative sampling at moderate hardness levels ( $\beta=5, 10$ ) improved downstream classification accuracy by over 40 percentage points compared to standard SCL and UCL.
NC Metrics: In supervised settings, increased hardness drove the model closer to NC. In unsupervised settings, an optimal hardness level existed ( $\beta=5$ ); higher hardness levels led to worse NC metrics and increased DC.
Initialization Sensitivity: Starting from a "near-NC" initialization, supervised settings (SCL/HSCL) maintained NC properties. Unsupervised settings (HUCL) with high hardness ( $\beta=10, 30$ ) drifted away from NC, exhibiting significant DC.
Batch Size: NC was achievable across batch sizes 64–512. However, very small batch sizes (32) hindered NC in high-cardinality datasets due to reduced negative pair diversity.

Significance and Claims

The paper claims to provide a simpler, compact, and transparent theoretical proof for the optimality of Neural Collapse in contrastive learning, removing the need for the conditional independence assumption on positive samples in the supervised setting.

The authors emphasize that while hard-negative sampling theoretically lowers the loss bound, its practical utility is nuanced:

It is essential for preventing Dimensional Collapse in Adam-optimized training, a phenomenon that standard CL often suffers from without normalization.
It significantly boosts downstream classification performance.
However, in unsupervised settings, excessive hardness can be detrimental, potentially causing class collisions that prevent the attainment of the optimal NC geometry.

The work concludes by identifying open problems, including deriving a tight lower bound for HUCL that accounts for class collisions and understanding the conditions under which DC can be avoided without necessarily achieving full Neural Collapse.