🤖 machine learning

Information theoretic underpinning of self-supervised learning by clustering

This paper establishes an information-theoretic foundation for self-supervised learning by formulating it as K-L divergence optimization, demonstrating that constraints on the teacher distribution theoretically justify common heuristics like batch centering and inverse cluster prior normalization to prevent mode collapse.

Original authors: Josef Kittler, Sara Atito, Muhammad Awais

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Josef Kittler, Sara Atito, Muhammad Awais

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of students (the AI) how to organize a massive, messy library of books, but you don't have any labels on the spines to tell you what genre each book belongs to. This is the challenge of Self-Supervised Learning (SSL): learning from data without a teacher telling you the answers.

For a long time, researchers have built very successful "organizers" using trial and error (heuristics). They found that if they made the students guess the genre, and then gently corrected them based on what the whole class thought, the students got really good at it. But nobody knew why this specific correction method worked so well.

This paper by Josef Kittler and colleagues is like a detective story. They didn't invent a new way to organize the library; instead, they went back to the math to explain why the existing methods work. They used a concept called Information Theory to prove that these "guess and correct" methods are actually solving a specific mathematical puzzle.

Here is the breakdown of their discovery using simple analogies:

1. The Student and the Teacher (Distillation)

In these AI systems, there are two networks working together:

The Student: The network trying to learn.
The Teacher: A network that acts as a guide, telling the student how to group the data.

Usually, in a classroom, the teacher has the answer key. But in this "self-supervised" library, the teacher doesn't know the answers either! The teacher has to guess the answers based on what the student is currently doing. They take turns: the student learns from the teacher's guess, then the teacher updates their guess based on the student's new performance. This is called alternating optimization.

2. The Problem: The "Lazy" Teacher (Mode Collapse)

If you let the teacher and student just guess freely, a problem called Mode Collapse happens. Imagine the teacher gets lazy and decides, "You know what? Let's just put every single book in the 'Mystery' pile."
The student learns this easily: "Oh, Mystery is the only category!" The student stops learning anything useful because everything looks the same. The AI has collapsed into a single, useless answer.

3. The Solution: The "Fairness" Rule

To stop the teacher from being lazy and putting everything in one pile, the authors introduced a mathematical rule (a constraint). They told the teacher: "You must distribute the books fairly across all the piles."

Mathematically, they used something called KL Divergence (a way to measure how different two guesses are). They added a "penalty" if the teacher tried to put too many books in one pile.

The Result: The teacher is forced to look at the data and say, "Okay, this book goes in 'Mystery', that one in 'Romance', and that one in 'Sci-Fi'."
The Magic Trick: To make this fairness rule work, the teacher has to adjust its own "confidence" in each pile. If a pile is empty, the teacher becomes very confident that a new book belongs there. If a pile is already full, the teacher becomes less confident. This is called scaling by inverse cluster priors.

4. The Big Discovery: Why "Centering" Works

Here is the most exciting part of the paper. The authors did some heavy math (using an inequality called Jensen's Inequality) to simplify their complex "Fairness Rule."

They discovered that this complex mathematical rule is actually very similar to a simple trick that engineers have been using for years called "Centering."

The Analogy: Imagine the library books are scattered on a floor. "Centering" is like telling everyone to stand up and move so that the average position of all the books is right in the middle of the room.
The Connection: The authors proved that mathematically, forcing the teacher to be fair (our complex rule) is almost the same thing as just moving the books to the center of the room (the simple trick).

This explains why the "Centering" trick works so well in popular AI systems (like DINO). It's not just a lucky guess; it's a simplified version of a deep mathematical principle that prevents the AI from getting lazy.

Summary

The Goal: Explain why current AI learning methods work without human labels.
The Method: They modeled the learning process as a Student and Teacher taking turns, with a rule to prevent the Teacher from being lazy (putting everything in one category).
The Finding: They proved that the complex math needed to keep the Teacher fair simplifies down to the simple, popular technique of "Centering" (moving data to the middle).
The Takeaway: This paper provides the "instruction manual" and the "why" behind the "how" that AI researchers have been using for years. It connects the messy, practical world of AI coding with the clean, logical world of mathematical theory.

Technical Summary: Information Theoretic Underpinning of Self-Supervised Learning by Clustering

Problem Statement
Self-supervised learning (SSL) has become a cornerstone for building foundation models, yet its theoretical underpinnings often lag behind its empirical success. While many SSL methods, particularly those based on clustering (e.g., DeepCluster, SwAV, DINO), rely on heuristic practices such as data centering and distillation, a rigorous mathematical justification for these procedures has been lacking. Specifically, the relationship between the optimization objectives used in SSL and the common practice of preventing "mode collapse" (where the model assigns all data to a single cluster) requires a formal information-theoretic explanation.

Methodology
The authors formulate SSL as a distillation learning problem, drawing an analogy to supervised learning but adapting it for unlabeled data. The core methodology involves the following steps:

K-L Divergence Formulation: The learning objective is defined as minimizing the Kullback-Leibler (K-L) divergence between a "teacher" distribution $Q(y|z)$ and a "student" distribution $P(y|z)$ . Unlike supervised learning where the teacher is fixed, in this SSL framework, both distributions must be learned.
Alternating Direction Optimization: The optimization is performed by alternating between:
- Student Update: Fixing the teacher distribution $Q$ and optimizing the student network parameters $\theta$ and cluster means $w_i$ to minimize the cross-entropy term of the K-L divergence.
- Teacher Update: Fixing the student distribution $P$ and optimizing the teacher distribution $Q$ .
Regularization for Mode Collapse: To prevent mode collapse, the authors introduce a regularization constraint based on the entropy of the cluster assignment priors. The goal is to maximize the entropy of the teacher's marginal distribution $Q(y)$ , ensuring data is distributed equitably across all $K$ clusters rather than collapsing into a few.
Derivation of Closed-Form Solution: By differentiating the regularized objective function with respect to the teacher distribution $Q(y|z)$ , the authors derive a closed-form solution. This solution reveals that the optimal teacher distribution is proportional to the student distribution scaled by the inverse of the cluster prior:
$Q(y|z) = \frac{P(y|z)}{Q(y)}$
where $Q(y)$ represents the prior probability of cluster $y$ .
Connection to Centering via Jensen's Inequality: The paper demonstrates that this inverse-prior normalization can be approximated using Jensen's inequality. By assuming the cluster weight vectors are centered (i.e., their mean is zero), the normalization term simplifies. This mathematical derivation shows that the popular heuristic of data centering (subtracting the batch mean from embeddings) is a theoretical approximation of the inverse-prior normalization derived from first principles.

Key Contributions

Theoretical Unification: The paper provides a unified information-theoretic framework that formulates SSL as K-L divergence optimization with alternating updates for student and teacher networks.
Justification of Heuristics: It offers a rigorous theoretical derivation for the inverse cluster prior normalization used to prevent mode collapse.
Link to Centering: It establishes a mathematical link between the derived inverse-prior normalization and the widely used centering procedure (subtracting the mean of embeddings) found in methods like DINO. This explains why centering is effective: it approximates the theoretically optimal inverse-prior scaling.
Closed-Form Teacher Update: The authors derive a specific update rule for the teacher distribution that explicitly incorporates cluster priors, moving beyond algorithmic heuristics (like Sinkhorn-Knopp) to a mathematical solution.

Results and Claims
The paper does not present new experimental benchmarks or propose a novel architecture. Instead, its "results" are theoretical derivations:

The authors show that minimizing the regularized K-L divergence naturally leads to a teacher update rule that scales the student probabilities by the inverse of the cluster priors.
Through the application of Jensen's inequality, they demonstrate that this scaling is mathematically approximated by centering the data (subtracting the mean embedding).
The paper claims that this derivation validates the effectiveness of existing successful SSL methods (like DINO) which utilize centering, confirming that these heuristics are not arbitrary but are grounded in information theory.

Significance
The significance of this work lies in its attempt to correct the "lopsided advancement" of AI, where experimental progress often outpaces theoretical understanding. By providing a theoretical underpinning for clustering-based SSL:

It validates the distillation and centering mechanisms used in state-of-the-art models, explaining why they work to prevent mode collapse.
It bridges the gap between the practical heuristics of SSL and the theoretical principles of information theory (specifically K-L divergence and entropy maximization).
It suggests future research directions, such as investigating the relationship between representation dimensionality and temperature, or exploring learning with fixed weight vectors, based on the derived model.

The authors maintain a modest stance, noting that while their model supports existing methods, the approximations (such as the use of Jensen's inequality) may have limitations in specific scenarios (e.g., large variations in embedding norms), though these do not invalidate the practical effectiveness of centering.