🤖 machine learning

Three Necessary Principles for Self-Supervised Visual Representation Learning

This paper argues that effective self-supervised visual representation learning requires the simultaneous integration of three non-overlapping principles—semantic invariance (observation), patch-level spatial prediction, and representational non-degeneracy (regularization)—to prevent model collapse and unify major methods under a single energy decomposition framework.

Original authors: Nikos Giakoumoglou, Paschalis Giakoumoglou, Tania Stathaki

Published 2026-08-11

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikos Giakoumoglou, Paschalis Giakoumoglou, Tania Stathaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to understand the world, but you have a strict rule: you cannot show it any labels. You can't say, "This is a cat," or "That is a car." This field of study is called self-supervised learning. The robot has to learn by looking at pictures of the world and figuring out patterns on its own, without a teacher telling it what's what.

To do this, the robot needs to build a mental map, or a "representation," of what it sees. Think of this like a library. If the robot just memorizes every single pixel of every photo, the library is a chaotic mess of dust and paper. But if it learns to group things by meaning—putting all "cats" on one shelf and all "cars" on another—it has learned a useful representation. The big question scientists have been asking is: What is the absolute minimum set of rules the robot needs to follow to build this library correctly without getting confused or giving up?

For a long time, researchers tried different tricks. Some told the robot to look at two different photos of the same thing and make sure they look similar (like recognizing a cat whether it's sleeping or playing). Others told the robot to look at a part of a photo and guess what the missing part looks like (like a puzzle). But often, the robot would get stuck. It would learn a "shortcut" where it just outputs the exact same boring answer for every single picture, effectively saying, "I don't care, everything is the same." This is called collapse, and it's the biggest failure mode in this field. The paper you are about to read dives deep into why these failures happen and proposes a new, three-part rulebook to stop them.

The Three-Ingredient Recipe for a Smart Robot

The authors of this paper, Nikos, Paschalis, and Tania, argue that to teach a robot to see without labels, you can't just rely on one or two tricks. You need three distinct ingredients working together. If you leave one out, the robot's brain will either break, become too simple, or lose its sense of space. They call these three principles Observation, Prediction, and Regularization.

Think of training a robot like training a detective to solve a mystery.

1. Observation: The "Same-Thing" Rule

The first principle is Observation. This is about teaching the robot that two different pictures of the same object are actually the same thing. If you take a photo of a dog in the morning and another in the afternoon, the robot needs to realize, "Hey, that's the same dog!"

In the paper's language, this is about semantic invariance. Imagine you are looking at a friend through a kaleidoscope. The colors shift, the shapes twist, but you still know it's your friend. The robot needs a "Projector" (a tool in its brain) that strips away the confusing twists and focuses on the core identity. If you only have this rule, the robot gets good at recognizing what things are, but it forgets where they are. It's like knowing a dog is a dog, but having no idea if the dog is sitting on the left or the right side of the room.

2. Prediction: The "Missing Piece" Game

The second principle is Prediction. This is about spatial structure. Imagine you cover part of a photo with a black box. The robot has to guess what's inside that box based on the rest of the picture.

This is the Prediction Principle. It forces the robot to understand how things fit together in space. If you only have this rule, the robot becomes a great puzzle solver for a single image, but it gets confused if you show it the same object from a different angle. It might think a cat seen from the side is a completely different creature than a cat seen from the front. It learns the "map" of one specific photo but doesn't learn that the map stays the same even if you rotate the photo.

3. Regularization: The "Anti-Cheat" Rule

The third principle is Regularization. This is the most critical part for preventing the robot from using shortcuts. Remember the "robot that gets stuck" that just outputs the same answer for everything? That's called collapse.

The authors prove that without a specific rule to stop this, the robot will find a way to use a shortcut. Imagine a student taking a test who realizes that if they just write "42" for every answer, they get a perfect score because the teacher's grading key is broken. The robot does the same thing: it finds a shortcut where it stops learning and just outputs a constant number.

The Regularization Principle is like a strict proctor who checks the student's work to make sure they aren't just guessing the same answer. It forces the robot's internal "library" to be full and diverse. It ensures that the robot uses all its brain power and doesn't collapse into a boring, flat line. The paper shows that if you try to use "negative examples" (showing the robot what not to match) to stop this shortcut, it's like trying to stop a leak with a band-aid; it helps a little, but eventually, the robot finds a way around it. You need a strong, explicit rule (a geometric constraint) to truly stop the collapse.

The Big Discovery: You Need All Three

The authors didn't just guess this; they built a mathematical framework (an "energy decomposition") that combines these three rules into one equation. They then ran experiments to see what happens when they remove one ingredient at a time.

Here is what they found:

If you drop Observation: The robot loses its ability to recognize that the same object looks different in different lights or angles. It becomes a spatial genius but a semantic idiot.
If you drop Prediction: The robot loses its sense of space. It knows what things are, but it can't tell you where they are or how they relate to each other. In their tests, the robot got much worse at finding specific patches of an image (like finding a specific leaf on a tree).
If you drop Regularization: The robot collapses. It stops learning entirely. The authors proved mathematically that without this rule, the robot can achieve a "perfect" score by just outputting the same boring number for every single image. It's a global failure.

They also looked at a popular trick used in many modern AI models called the Momentum Encoder. This is a technique where the robot uses a "slow-moving" version of itself to help learn. The authors showed that while this trick helps the robot learn faster at the beginning, it doesn't actually stop the robot from collapsing in the long run. Once the robot settles down, the momentum encoder just copies whatever the main robot is doing. If the main robot is collapsed, the momentum one is too. The only thing that truly guarantees the robot stays smart is that explicit Regularization rule.

Why This Matters

The paper unifies almost every major method in self-supervised learning. Whether a method is called "Contrastive" (like SimCLR) or "Predictive" (like JEPA), the authors show that they are all just different ways of mixing these three ingredients. Some methods mix Observation and Regularization but skip Prediction. Others mix Prediction and Regularization but skip Observation.

The authors' experiments, run on a small dataset called STL-10 with a model called ViT-Tiny, showed that when you combine all three, you get the best results. Specifically, the model that had all three principles achieved a 55.0% accuracy in a standard test, while models missing one principle dropped significantly (for example, missing Observation dropped accuracy by 13.4 points).

They also proved that these three rules don't fight each other. The "Observation" rule sends a uniform signal across the whole image, while the "Prediction" rule sends a specific signal only to the parts of the image being guessed. They work like a team, not a tug-of-war.

The Takeaway

The main message is simple but powerful: You cannot build a robust, smart visual AI by relying on just one type of trick. You need a system that observes sameness, predicts missing parts, and is strictly regularized to prevent shortcuts.

The authors suggest that future AI designers should stop trying to invent complex, hidden tricks to stop their models from collapsing. Instead, they should explicitly build these three principles into their systems. It's like building a house: you need a foundation (Regularization), walls (Observation), and a roof (Prediction). If you leave out the foundation, the whole thing falls down. If you leave out the roof, the inside gets wet. And if you leave out the walls, you don't have a house at all.

This paper doesn't claim to have built the biggest or fastest AI ever. Instead, it provides a clear, mathematical map of why AI works the way it does, proving that these three principles are not just helpful suggestions, but absolutely necessary conditions for a robot to learn to see the world without a teacher.

Technical Summary: Three Necessary Principles for Self-Supervised Visual Representation Learning

Problem Statement

Self-supervised learning (SSL) for visual representation learning is currently fragmented into distinct families of methods, primarily invariance-based approaches (e.g., SimCLR, BYOL) and Joint-Embedding Predictive Architectures (JEPAs) (e.g., I-JEPA). While both aim to learn compatibility functions over pairs of observations, they suffer from a common failure mode: representational collapse, where the encoder maps all inputs to a constant or low-dimensional subspace.

Existing methods address collapse through various mechanisms—negative mining, asymmetric dynamics (momentum encoders), or statistical regularization—but these are often conflated with the primary learning objective. The paper argues that the field lacks a unified theoretical framework that explicitly separates the necessary components of a complete training signal. Specifically, it is unclear whether current methods satisfy all necessary conditions or if implicit mechanisms (like momentum encoders) provide genuine guarantees against collapse at convergence.

Methodology: The Unified Energy Decomposition

The authors propose a unified Energy-Based Model (EBM) framework where the total training energy $F_w(x, y)$ is decomposed into three non-overlapping terms:

$F_w(x, y) = \alpha F_{proj}(x, y) + \beta F_{pred}(x, y) + \gamma \Omega(f_\theta)$

This decomposition formalizes three distinct principles:

Observation Principle ( $F_{proj}$ ): The encoder must produce semantically invariant global representations. This is achieved via a projector $h_\psi$ that maps globally pooled representations to an embedding space where view-level compatibility is measured (e.g., via NT-Xent or MSE). This principle ensures cross-view semantic alignment but discards spatial structure.
Prediction Principle ( $F_{pred}$ ): The encoder must support latent-space prediction of masked content. A predictor $g_\phi$ maps context patches and positional information to predicted target representations. This principle enforces spatial consistency within an image but does not inherently enforce cross-view invariance.
Regularization Principle ( $\Omega$ ): The training objective must contain a term that prevents dimensional collapse by constraining the geometry of the batch distribution of representations. This term must be strictly positive at any collapsed solution (Dirac mass) and minimized only when the representation covariance has full effective rank. Examples studied include VCReg, SIGReg, and RDMReg.

The authors argue that dropping any single principle admits degenerate solutions that the remaining two cannot rule out. They prove that in the absence of explicit regularization, combining observation and prediction admits a constant encoder as a global minimizer under negative-free alignment. Furthermore, they demonstrate that contrastive alignment (observation) provides only "self-limiting" resistance to collapse due to gradient decay as representations homogenize.

Key Contributions

Formalization of Three Principles: The paper defines Observation, Prediction, and Regularization as non-overlapping conditions on the energy surface. It proves that regularization is optimization-theoretically necessary in negative-free regimes and that contrastive resistance is self-limiting.
Structural Necessity: The authors provide structural arguments showing that prediction is required for spatial signals and observation is required for cross-view semantic alignment. They prove that these objectives are gradient-complementary at the encoder output: observation provides a uniform global gradient, while prediction provides a spatially local gradient concentrated on context patches.
Unification of Prior Work: The paper demonstrates that every major SSL method is a special case of this unified energy decomposition. By setting coefficients $\alpha, \beta, \gamma$ to zero, one recovers invariance methods, predictive methods, or hybrid approaches.
Momentum Encoder Redundancy: The authors prove that if an online encoder converges to a fixed point, the momentum target encoder converges to the same point. Consequently, the momentum encoder provides no collapse guarantee beyond what an explicit regularizer offers; its benefit is limited to transient target smoothing.
Unified Taxonomy: A comprehensive taxonomy (Table 5) classifies existing methods based on which of the three principles they satisfy, revealing that many state-of-the-art methods (e.g., DINOv2, C-JEPA) satisfy all three but retain redundant implicit mechanisms.

Experimental Results

The authors validate their theoretical claims through controlled experiments on ViT-Tiny trained on STL-10.

Collapse Verification: Experiments confirm that combining observation and prediction without regularization leads to dimensional collapse (low effective rank) and poor linear-probe accuracy (38.7% vs. 49.2% with regularization).
Necessity of Principles:
- Removing Regularization (while keeping Observation and Prediction) results in a 3.4-point drop in accuracy and a collapse in effective rank.
- Removing Prediction results in a 5.5-point drop and a significant degradation in patch-level spatial retrieval (spatial recall@5 drops from 93.0% to 89.9%), confirming the lack of spatial training signals.
- Removing Observation causes the largest single accuracy drop (13.4 points), confirming the necessity of cross-view semantic alignment.
Gradient Complementarity: Gradient analysis shows that the invariance objective produces a uniform gradient across all patch tokens, while the prediction objective produces gradients only on visible context tokens, confirming they act on disjoint aspects of the representation.
Momentum Redundancy: Experiments show that models with and without momentum encoders converge to the same asymptotic accuracy when an explicit regularizer is present, validating the theoretical claim of redundancy at convergence.
Regularizer Interchangeability: Different explicit regularizers (SIGReg, RDMReg) yield comparable performance, whereas VCReg shows lower accuracy despite higher effective rank, attributed to gradient conflict with the prediction objective.

Significance and Claims

The paper claims to provide a design discipline for self-supervised learning by isolating the specific roles of observation, prediction, and regularization. The significance lies in:

Decoupling Collapse Prevention: Moving away from entangled implicit mechanisms (like momentum encoders or centering) toward explicit geometric regularization, which offers a provable guarantee against collapse at convergence.
Modularity: The decomposition allows practitioners to specify, swap, and diagnose objectives independently. A failure can be attributed to a missing principle rather than an opaque interaction of architectural tricks.
Theoretical Clarity: The work clarifies that the "three principles" are structurally necessary and non-substitutable at the studied scale, and that the field's diverse methods are merely different instantiations of a single energy decomposition.

The authors maintain modesty, noting that their validation is at a small scale (ViT-Tiny/STL-10) and that whether these ordering and interchangeability properties persist at larger scales remains an open question. They frame their contribution as an analytical framework rather than a new state-of-the-art system.