🤖 machine learning

The role of class encoding in neural collapse

This paper investigates how label encoding influences neural collapse, demonstrating that for one-hot encoded labels on balanced data, increasing bias regularization causes uncentered mean features to transition from a simplex equiangular tight frame to an orthogonal frame, while also showing that the classifier's bias generally serves to center arbitrary label encodings.

Original authors: Bastien Massion, Roy Makhlouf, Estelle Massart

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bastien Massion, Roy Makhlouf, Estelle Massart

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a team of robots (a neural network) to sort a pile of mixed-up toys into different boxes. The robots have a "brain" (the hidden layers) that processes the toys, and a "decision maker" (the final classifier) that points to the correct box.

This paper investigates a strange and beautiful phenomenon called Neural Collapse. When the robots get really good at their job (making zero mistakes), something magical happens in their brains: the internal representations of toys from the same box start to clump together into tight, perfect groups.

Here is the simple breakdown of what the authors discovered, using everyday analogies:

1. The Two Shapes of Organization

The paper looks at how these clumps of toys arrange themselves. They found that the arrangement depends heavily on two things: how the boxes are labeled and how much the decision maker is allowed to "shift" its position.

The "Orthonormal" Shape (The Grid): Imagine the robots arrange the toy groups like the corners of a perfect square grid. Every group is at a right angle to the others. This happens when the decision maker is forced to stay exactly at the center (zero bias).
The "Simplex" Shape (The Pyramid): Imagine the groups arrange themselves like the corners of a perfect pyramid (or a triangle if there are only three groups). They are all equally spaced from each other, but they are tilted slightly. This happens when the decision maker is allowed to move freely.

The Big Discovery: The authors realized these two shapes aren't actually different worlds. They are just the same shape, just shifted.

Think of the "Grid" as a pyramid sitting on a table.
If you lift the whole pyramid up (add a "bias"), the center of the pyramid moves, but the relative angles between the corners stay the same.
The paper shows that by adjusting a "regularization knob" (which controls how much the decision maker is allowed to move), you can smoothly slide the robots' brain from the Grid shape to the Pyramid shape.

2. The Role of the "Label" (The Box Tags)

The paper asks: Does it matter how we write the names on the boxes?

Standard Labels (One-Hot): Usually, we label boxes with a "1" for the right box and "0" for everything else (like a light switch).
The Bias's Job: The authors found that the decision maker's "bias" acts like a compensator. Its main job is to make sure the average position of all the labels sits right in the middle of the room.
- If your labels are naturally centered (balanced), the bias stays at zero.
- If your labels are off-center (like if you use a weird coding system), the bias shifts to pull the whole system back to the center. It's like a tightrope walker adjusting their pole to stay balanced.

3. What Changes and What Stays the Same?

The paper checks if changing the labels (the encoding) breaks the magic of Neural Collapse.

The "Clumping" (NC1): This part is super robust. No matter how you label the boxes or how you tune the bias, the robots will always clump the same toys together. This is like saying, "No matter how you paint the boxes, the red cars will always park in the red zone."
The "Perfect Symmetry" (NC2): This is where the labels matter. The perfect pyramid shape only forms if the labels themselves have a certain symmetry. If you scramble the labels in a weird way, the robots might not form that perfect pyramid, though they will still clump.
The "Mirror Image" (NC3): Usually, the decision maker's weights look exactly like the clumped toy groups (a property called self-duality). The paper found that if you rotate or flip the labels, this mirror image breaks. However, it's not really broken; it's just rotated. If you rotate the decision maker back, the mirror image returns. So, the connection is still there, just wearing a different hat.

Summary

In short, this paper explains that Neural Collapse is a very stable phenomenon.

The robots always group similar things together.
The specific geometric shape of these groups (Grid vs. Pyramid) is controlled by a "bias knob" and the way the labels are written.
The bias acts as a centering mechanism, ensuring the system stays balanced regardless of how the labels are encoded.
Even if the labels are scrambled, the fundamental "clumping" remains, and the relationship between the groups and the decision maker is just rotated, not destroyed.

The authors didn't propose new medical uses or future applications; they simply mapped out the mathematical "geometry" of how these AI brains organize themselves when they are doing their best work.

Technical Summary: The Role of Class Encoding in Neural Collapse

Problem Statement
Neural Collapse (NC) is a phenomenon observed in the terminal stage of training deep neural network classifiers, characterized by specific geometric structures in the last-hidden-layer activations when training error reaches zero. While NC has been extensively studied under the Unconstrained Features Model (UFM) with Cross-Entropy loss, its behavior under Mean Squared Error (MSE) loss remains a subject of investigation. Specifically, prior work has identified two distinct structural outcomes for class mean activations in the UFM with MSE loss and one-hot encoded labels: an Orthogonal Frame (OF) when the classifier bias is absent or heavily regularized, and a Simplex Equiangular Tight Frame (ETF) when an unregularized bias is present. The relationship between these structures and the influence of arbitrary label encodings on NC properties are not fully understood. This paper addresses the role of label encoding and the bias term in the emergence of these geometric structures.

Methodology
The authors analyze the problem using the Unconstrained Features Model (UFM) with an MSE training loss. The optimization problem involves jointly minimizing the loss over unconstrained features ( $H$ ), a linear classifier ( $W$ ), and a bias term ( $b$ ), subject to $\ell_2$ regularization on all parameters. The study considers:

Label Encodings: From standard one-hot encoding to arbitrary encoding matrices $\bar{Y} \in \mathbb{R}^{K \times K}$ , including label smoothing.
Bias Regularization: The impact of varying the regularization coefficient $\lambda_b$ associated with the classifier bias.
Data Balance: Analysis covers both balanced and imbalanced data scenarios.

Theoretical derivations rely on first-order optimality conditions to characterize global minimizers. The authors introduce the concept of a "Shifted Simplex Equiangular Tight Frame" (SSETF) to unify the observed geometric structures.

Key Contributions and Results

Unification of OF and Simplex ETF Structures:
The paper demonstrates that the transition between an Orthogonal Frame (OF) and a Simplex ETF is governed by the bias term. For one-hot encoded labels and balanced data, as the bias regularization coefficient $\lambda_b$ decreases, the uncentered mean features transition continuously from an OF to a Simplex ETF.
- Mechanism: The optimal bias $b^*$ acts to compensate for the non-centering of the labels. In the limit of unregularized bias ( $\lambda_b = 0$ ), the bias equals the mean of the labels, effectively centering the problem and yielding a Simplex ETF structure.
- SSETF Definition: The authors define a Shifted Simplex ETF (SSETF) as a generalization that encompasses both OFs (shift parameter $\beta=0$ ) and Simplex ETFs (shift parameter $\beta=1$ ). They prove that centering any SSETF results in a Simplex ETF.
Role of Bias in Arbitrary Encodings:
Extending beyond one-hot encoding, the authors show that for any arbitrary label encoding and imbalanced data, the optimal bias in the UFM aims to center the labels. Specifically, if the labels are centered ( $Y \mathbf{1}_N = 0$ ), the optimal bias is zero regardless of $\lambda_b$ . If labels are not centered, the bias term shifts the solution to compensate for the discrepancy between the global mean of the labels and the origin.
Robustness of Variability Collapse (NC1):
The paper proves that the first property of Neural Collapse (NC1)—the collapse of features to their class means—holds for any label encoding and any bias regularization strength $\lambda_b \geq 0$ , provided the data is balanced. This indicates that variability collapse is independent of the specific encoding scheme.
Impact on Self-Duality (NC3) and NC3':
The study reveals that the standard self-duality property (NC3), where classifier weights align with class means, is not robust to arbitrary encodings. Applying an orthogonal transformation to the labels breaks the direct alignment between $W^*$ and the class means $\bar{H}^*$ .
- Proposed Solution: The authors propose a weaker variant, NC3', which states that classifier weights converge to class means up to rescaling and an orthogonal rotation/reflection. They prove that NC3' holds for any balanced encoding under the UFM with MSE loss.
Convergence to Simplex ETF (NC2) and SSETF:
The paper establishes that if the optimal mean features form an SSETF, their centered counterparts form a Simplex ETF (satisfying NC2). The authors conjecture that for balanced scenarios, the mean features form an SSETF if and only if the label encoding itself is an SSETF. This family includes both one-hot encoding and label smoothing.

Significance and Claims
The paper claims to provide a theoretical bridge between the previously distinct OF and Simplex ETF structures observed in MSE-trained classifiers, attributing the difference solely to the bias term's role in centering the labels. By introducing the SSETF framework, the work generalizes the understanding of Neural Collapse to arbitrary label encodings.

The authors modestly note that while they have identified sufficient conditions for NC2 (via SSETF encodings) and proven the robustness of NC1 and NC3', the complete characterization of which encodings lead to NC2 in imbalanced settings remains an open question. The work does not propose new algorithms or experimental applications but rather offers a theoretical clarification of how label encoding and bias regularization interact to shape the geometric landscape of neural collapse.

1. The Two Shapes of Organization

2. The Role of the "Label" (The Box Tags)

3. What Changes and What Stays the Same?

Summary

More like this