🤖 machine learning

IdEst: Assessing Self-Supervised Learning Representations via Intrinsic Dimension

This paper introduces IdEst, a computationally efficient method that estimates the intrinsic dimension of self-supervised learning representations using a Minimum Spanning Tree dimension estimator, demonstrating that this geometric metric strongly correlates with downstream linear probe performance and serves as a principled alternative for hyperparameter selection.

Original authors: Julie Mordacq, Vicky Kalogeiton, Steve Oudot

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Julie Mordacq, Vicky Kalogeiton, Steve Oudot

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of unlabeled books (photos, videos, etc.). You want to teach a robot to understand these books without a librarian telling it what each one is about. This is called Self-Supervised Learning (SSL). The robot reads the books, groups similar ones together, and builds a mental map of the world.

But here's the problem: How do you know if the robot's mental map is any good?

Usually, to check the robot, you have to hire a human librarian to give it a test (a "linear probe"). You ask the robot to sort the books into specific categories (like "cats" vs. "dogs"). If it gets a high score, the map is good. But this is slow, expensive, and requires a lot of human effort.

Enter IDEST.

The authors of this paper propose a new way to check the robot's map without hiring a librarian or looking at the labels. They call their method IDEST (which sounds like the Latin phrase id est, meaning "that is").

The Core Idea: The "Crowded Room" Analogy

Think of the robot's mental map as a room where it places every book it has ever seen.

A bad map: The robot throws all the books into a huge, empty warehouse. Everything is scattered, and the room feels chaotic and empty. It's hard to find patterns.
A good map: The robot organizes the books into a cozy, well-structured living room. Similar books are close together, and the space is used efficiently.

The authors discovered a simple rule: The more efficiently the robot packs the books into its mental room, the better it will be at future tasks.

They measure this efficiency using something called Intrinsic Dimension (ID).

Imagine a flat sheet of paper (2D) floating in a 3D room. Even though the room is 3D, the paper only needs 2 dimensions to describe it.
If the robot's map is "flat" and organized (low dimension), it's efficient.
If the map is "bubbly" and messy, stretching out into every corner of the room (high dimension), it's inefficient.

The Problem with Old Tools

Previously, scientists tried to measure this "room efficiency" using tools like TwoNN or MLE.

The Analogy: Imagine trying to measure the shape of a room by dropping a few marbles on the floor and seeing how they bounce.
The Flaw: In the messy, high-tech world of modern AI, these tools are like trying to measure a room with a ruler made of jelly. They get confused by noise, they break if the marbles aren't perfectly spaced, and they often give wild, wrong answers (like saying a 1D line is actually infinite).

The New Tool: The "String" Method (IDEST)

The authors used a different tool called dimMST (Minimum Spanning Tree dimension estimator).

The Analogy: Instead of dropping marbles, imagine you have a string. You connect every book in the robot's room to its nearest neighbor with the string, creating a giant web that touches every single book without any loops.
Why it works: You then measure the total length of that string. By looking at how the string length grows as you add more books, you can mathematically figure out the true "shape" of the room.
The Benefit: This "string" method is tough. It doesn't care if the books are noisy, clumped together, or spread out. It gives a reliable measurement even in messy conditions.

What They Found

The authors tested IDEST on many different AI models (like DINO, CLIP, and VICReg) and many different datasets (like ImageNet).

The Golden Rule: They found a strong, consistent link: The lower the "Intrinsic Dimension" (the more efficient the room), the higher the robot's test scores.
- If IDEST says the dimension is low, the robot is likely smart and ready for work.
- If IDEST says the dimension is high, the robot is likely confused.
- Note: The correlation was so strong that they could predict how well a model would do on a test just by measuring the "string length" of its map, without ever running the test.
Watching the Robot Learn: They watched the robot learn over time. As the robot got better at its job, the "string length" (dimension) naturally shrank. IDEST could track this improvement in real-time, acting like a fitness tracker for the AI's brain.
Saving Time and Money: Because IDEST doesn't need labels (it's "unsupervised"), researchers can use it to pick the best settings for their AI models without running expensive tests. It's like a mechanic checking the engine's vibration to tune the car, rather than driving it on a race track to see how fast it goes.

Summary

The paper introduces IDEST, a new, cheap, and fast way to judge if a self-supervised AI model is good. Instead of testing the model with human-labeled data, IDEST measures how "tightly packed" and organized the AI's internal map is.

Low Dimension (Tight packing) = Good Model.
High Dimension (Messy packing) = Bad Model.

It works like a "string" that connects the dots in the AI's mind, revealing the true shape of its knowledge without needing a single label. This allows scientists to build better AI models faster and cheaper.

Technical Summary: IDEST – Assessing Self-Supervised Learning Representations via Intrinsic Dimension

Problem Statement

Self-supervised learning (SSL) has become a dominant paradigm for extracting meaningful representations from unlabeled data. However, the standard protocol for evaluating these representations—linear probing—presents significant limitations. It is computationally expensive, sensitive to hyperparameters, and offers limited insight into the geometric structure of the representation space. Furthermore, existing geometric proxies for representation quality, such as those based on spectral properties (e.g., $\alpha$ -ReQ, RankMe), often rely on assumptions (like power-law decay) that fail under representation collapse or are restricted to specific architectures (e.g., Joint-Embedding Architectures). Additionally, while Intrinsic Dimension (ID) has been linked to generalization in supervised convolutional networks, its behavior and utility in modern SSL settings remain largely unexplored, particularly given the unique challenges of SSL data distributions (non-asymptotic regimes where $n \approx d$ and dependencies between augmented views).

Methodology: IDEST

The authors propose IDEST (Intrinsic Dimension Estimation for SSL using minimum spanning Trees), an unsupervised criterion for assessing SSL representations based on estimating their Intrinsic Dimension.

Theoretical Foundation

The method is grounded in the connection between generalization error and intrinsic dimension. Citing Konz & Mazurowski (2024), the generalization error $L$ for a well-trained model interpolating a dataset of size $N_D$ scales as $O(N_D^{-1/d})$ , where $d$ is the intrinsic dimension. This implies that for a fixed dataset size, a lower intrinsic dimension corresponds to a more efficient representation and potentially better downstream performance.

Estimator Selection: dimMST

To estimate $d$ , the authors reject standard parametric estimators like TwoNN and Maximum Likelihood Estimation (MLE). These methods assume data points are sampled i.i.d. from a distribution with locally constant density and often fail in SSL settings due to noise, sampling density variations, and the specific dependencies introduced by data augmentations (as demonstrated in Figure 2, where TwoNN diverges on a simple helix manifold).

Instead, IDEST utilizes the Minimum Spanning Tree (MST) dimension estimator ( $\text{dimMST}$ ), derived from the theory of Euclidean functionals (Costa & Hero, 2006).

Mechanism: The estimator relies on the asymptotic growth rate of the total length of the MST of a point cloud. The relationship is defined as:
$\lim_{n \to \infty} n^{-\frac{d-1}{d}} \cdot L(\text{MST}(X_n)) = C'$
where $L$ is the total edge length and $C'$ is a constant.
Implementation: The intrinsic dimension $d$ is estimated via log-log linear regression on subsamples of increasing size $n_i$ :
$\log(L(\text{MST}(X_{n_i}))) \approx \frac{d-1}{d} \log(n_i) + \log(C)$
The slope $m$ yields $d = 1/(1-m)$ .
Robustness: $\text{dimMST}$ is theoretically robust to noise, sampling density variations, and high ambient dimensionality, making it suitable for the $n \approx d$ regime inherent in vision SSL.

Evaluation Protocol

IDEST operates on frozen representations passed to the classifier head, consistent with standard linear probing protocols. For models without a class token (e.g., I-JEPA), patch tokens are average-pooled; for models with a [CLS] token (e.g., DINO), the token is used directly.

Key Contributions

Proposal of IDEST: An unsupervised criterion for evaluating SSL representations based on intrinsic dimension estimation via MST.
Empirical Validation: Demonstration that IDEST strongly correlates with downstream linear probe performance across diverse datasets, architectures, and SSL pretraining objectives.
Hyperparameter Selection: Evidence that IDEST enables efficient, label-free hyperparameter selection, significantly reducing computational costs compared to supervised alternatives.

Experimental Results

1. Correlation with Downstream Performance

The authors evaluated 33 models across 14 SSL methods (including VICReg, DINO, DINOv2, I-JEPA, CLIP, SigLIP, etc.) and various architectures (ResNet, ViT).

Intra-Dataset Correlation: IDEST shows a strong negative correlation with Top-1 linear probe accuracy on ImageNet, iNat-18, iNat-21, and SUN397. Spearman's $\rho \approx -0.8$ and Kendall's $\tau \approx -0.6$ indicate that models with lower intrinsic dimensions consistently achieve higher accuracy.
Inter-Dataset Correlation: IDEST computed on ImageNet serves as a reliable proxy for performance on other datasets (iNat-18, iNat-21, CIFAR-10, CIFAR-100), suggesting it captures intrinsic properties of the representation rather than dataset-specific artifacts.
Alternative Protocols: The correlation holds under k-NN evaluation and on the ImageNet-v2 validation set.

2. Training Dynamics

Offline Probing: IDEST tracks the evolution of representation quality throughout pretraining. As training epochs increase and linear probing accuracy improves, the estimated intrinsic dimension consistently decreases.
Online Probing: IDEST effectively monitors downstream performance improvements even when a classifier is trained jointly during pretraining (without backpropagating to the encoder), provided the initial training stages are surpassed.

3. Unsupervised Hyperparameter Selection

The authors used IDEST to select optimal hyperparameters (learning rate, weight decay, temperature, etc.) for VICReg, DINO, and I-JEPA.

Performance: IDEST-selected models achieved performance competitive with or superior to other unsupervised baselines ( $\alpha$ -ReQ, RankMe) and a weakly-supervised baseline (LiDAR).
Efficiency: Selecting models via IDEST avoids the computational cost of training linear probes for every hyperparameter configuration.

Significance and Claims

The paper claims that intrinsic dimensionality serves as a principled geometric proxy for assessing SSL representations. By leveraging the MST dimension estimator, IDEST provides a robust, label-free metric that:

Complements standard supervised probing protocols by offering insights into the geometric structure of the representation space.
Generalizes across diverse SSL paradigms (joint-embedding, joint-predictive, vision-language) and architectures.
Reduces Cost by enabling efficient hyperparameter selection without the need for labeled data or expensive linear probing evaluations.

The authors emphasize that their work underscores the promise of geometric descriptors as a fundamental tool for understanding and optimizing self-supervised learning, moving beyond separability to consider how efficiently information is compressed into low-dimensional geometric structures.