🤖 machine learning

A Comparative Study of Label-free Representation Quality Metrics in Deep Learning

This paper presents a comprehensive comparative study of label-free representation quality metrics across 260 vision models, identifying intrinsic dimensionality as the most reliable predictor while demonstrating that the effectiveness of all metrics is significantly moderated by architecture class and training objectives.

Original authors: Daniel Richards Arputharaj, Daniel Jönsson, Gabriel Eilertsen

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Richards Arputharaj, Daniel Jönsson, Gabriel Eilertsen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern artificial intelligence, a quiet revolution has taken place. For years, the most powerful computer vision systems were built by feeding them millions of labeled images, teaching them to recognize a cat by showing them a thousand pictures of cats with the word "cat" written underneath. But a newer, more efficient approach has emerged, where models learn to understand the world by looking at images without any labels at all. These systems, often called self-supervised, teach themselves by spotting patterns, similarities, and differences within the data itself. The result is a library of thousands of pre-trained models, ready to be downloaded and used as the eyes for new applications. Yet, this abundance has created a new problem: how does a developer choose the right model? With no labels to test against, and no time to run expensive training experiments on every candidate, there is a desperate need for a quick, reliable way to judge the quality of a model's internal understanding just by looking at its output.

This is the challenge tackled by a team of researchers at Linköping University, who set out to test a collection of tools designed to measure the quality of these label-free models. They examined two hundred and sixty different vision models, ranging from standard image classifiers to those trained with advanced self-supervised techniques, and tested them across six different datasets covering everything from generic objects like cars and animals to fine-grained details like specific flower species and scenes from satellite imagery. The researchers wanted to know if the existing mathematical shortcuts used to judge these models actually worked, or if they were merely giving false confidence. They grouped the available tools into three families: those that look at the overall spread of data, those that examine how points relate to one another, and those that measure the complexity of the shape the data forms. By running controlled experiments and comparing the results against actual task performance, they discovered that while some tools are useful in specific situations, one particular measure stands out as the most trustworthy guide for the general public.

The researchers began by categorizing the various ways to measure a model's internal state without using labels. Some methods, which they called spectral metrics, analyze the distribution of energy or variance across the model's internal dimensions, essentially checking if the model is using its full capacity or collapsing into a few narrow directions. Others, known as relational metrics, look at the distances between different data points to see if the model has organized them into a useful structure. A third group, manifold-based metrics, attempts to estimate the true complexity or dimensionality of the shape that the data occupies. The team first tested these tools on synthetic data they generated in the lab, carefully controlling the shape of the data to see how each tool reacted. They found that the spectral tools were highly sensitive to the specific shape of the data distribution; some would jump to high values when the data was perfectly uniform, while others would spike when the data was concentrated. This revealed that no single spectral tool could be trusted to give a consistent reading across all possible scenarios, as they were often measuring different things entirely.

When the researchers moved to real-world models, the picture became even more nuanced. They evaluated the tools against the actual accuracy of the models when applied to new tasks. The results showed that the reliability of these tools was not a fixed property but depended heavily on the type of model and how it was trained. For instance, tools designed to measure how evenly data is spread out worked well for models trained with self-supervised methods, which are explicitly taught to spread their representations out to avoid collapsing. However, these same tools failed completely when applied to models trained with traditional labeled data, where the goal is often to cluster similar items tightly together. Similarly, tools that measured the stability of the mathematical system behind the model were found to be misleading, as their scores were often just a reflection of the mathematical solver used rather than the quality of the model itself.

Amidst this variability, one metric emerged as a clear and consistent leader: intrinsic dimensionality. This measure estimates the minimum number of parameters needed to describe the data, essentially asking how many "directions" the data actually needs to exist in. The researchers found that this tool was the most reliable predictor of success across almost all scenarios. When the intrinsic dimensionality was lower, the models tended to perform better on downstream tasks. This held true whether the model was a standard convolutional network or a modern transformer, and whether it was trained with labels or without. The only time this rule weakened was when the task moved away from recognizing distinct objects, such as when identifying specific scenes or remote-sensing images, where the relationship between the data's shape and the task became less direct.

The study also highlighted a critical lesson for anyone trying to select a model: the context matters immensely. A tool that works perfectly for one family of models might be useless for another. For example, a metric that suggested a model was high-quality because it had a high "effective rank" might be completely wrong if the model was of a different architecture. The researchers demonstrated that grouping models by their architecture class or training objective was essential to getting a clear signal. Without this careful sorting, the results were often contradictory, with different tools pointing in opposite directions. The team concluded that while the field of label-free evaluation is promising, it requires a more sophisticated approach than simply picking the highest number. The most robust path forward is to rely on the measure of intrinsic dimensionality, which consistently tracks the quality of the representation, while treating other metrics with caution and understanding exactly what specific conditions they are designed for.

Ultimately, this work provides a clearer map for navigating the growing forest of pre-trained models. It suggests that the quest for a single, universal "score" for model quality may be complex, as different models organize their knowledge in fundamentally different ways. Instead, the most reliable indicator of a model's potential is its ability to compress information into a lower-dimensional, efficient structure. For developers and researchers, this means that when choosing a model without the luxury of testing it on the final task, looking at the intrinsic dimensionality offers the best chance of making the right choice. The study does not claim to have solved the problem of model selection, but it has provided a much-needed filter, separating the tools that offer genuine insight from those that merely reflect the quirks of the model's training or architecture.

Technical Summary: A Comparative Study of Label-free Representation Quality Metrics in Deep Learning

Problem Statement

The proliferation of pre-trained deep neural networks, particularly through self-supervised learning (SSL) and public repositories like HuggingFace, has created a challenge in selecting optimal feature extractors for downstream tasks without extensive computational overhead. Traditionally, model suitability is evaluated via label-dependent methods such as linear probing, K-nearest neighbors (KNN), or full fine-tuning. However, as the number of available models grows, exhaustive search becomes computationally prohibitive.

This has motivated the development of label-free metrics, which assess representation quality solely based on the geometry of the feature space (e.g., covariance structure, pairwise relations, or manifold geometry) without requiring access to target labels. While various metrics have been proposed (e.g., RankMe, $\alpha$ -ReQ, Intrinsic Dimension), existing evaluations often validate single metrics against narrow architectural or training configurations. Consequently, it remains unclear which metrics are reliable across diverse architectures, training objectives (supervised vs. SSL), and datasets, or how these metrics differ in their sensitivity to spectral shapes and representation dimensions.

Methodology

The authors conduct a comprehensive comparative study involving 260 vision models across six datasets (spanning generic object classification, fine-grained classification, scene recognition, and geospatial tasks). The methodology is structured into three main phases:

Taxonomy and Unification:
The paper categorizes existing label-free metrics into three families based on their construction:
- Spectral Metrics: Based on the eigenvalues of the representation covariance matrix or singular values of the representation matrix (e.g., $\alpha$ -ReQ, RankMe, NE Sum, Condition Number $\kappa$ ).
- Relational Metrics: Based on pairwise relationships between samples (e.g., Self-Cluster, Diffusion Spectral Entropy (DSE)).
- Manifold-based Metrics: Based on the local geometry and dimensionality of the representation manifold (e.g., Intrinsic Dimension (ID)).
  The authors reformulate these metrics where possible to highlight theoretical connections and unify their frameworks.
Synthetic Sensitivity Analysis:
To decouple the effects of architecture and dataset from spectral properties, the authors generate synthetic eigenspectra following a power-law decay ( $\lambda_i \propto i^{-\alpha}$ ). They systematically vary the decay exponent $\alpha$ and dimensionality $d$ to characterize how spectral metrics respond to changes in spectral shape and representation dimension.
Empirical Correlation Study:
The authors evaluate the correlation between each metric and downstream task accuracy.
- Downstream Proxies: They use a closed-form Ordinary Least Squares (OLS) linear probe and a KNN classifier ( $k=1$ ) as proxies for transfer performance.
- Stratification: Results are stratified by architecture class (ViT, ResNet, ConvNeXT, RegNet) and training objective (Supervised vs. SSL).
- Statistical Rigor: Spearman's rank correlation is computed using a cluster-robust bootstrap procedure to account for dependencies within model families, with Benjamini–Hochberg correction for multiple comparisons.

Key Contributions

A Unified Taxonomy: The paper establishes a taxonomy grouping label-free metrics into Spectral, Relational, and Manifold-based families, providing reformulated definitions that clarify similarities and differences (e.g., showing the analytical link between Self-Cluster and DSE).
Sensitivity Characterization: Through controlled synthetic experiments, the authors demonstrate that spectral metrics exhibit distinct sensitivities to spectral shape ( $\alpha$ ) and dimensionality ( $d$ ). For instance, RankMe and NE Sum are maximized at uniform spectra ( $\alpha=0$ ), while the condition number $\kappa$ is most sensitive at spectral extremes.
Systematic Reliability Assessment: The study provides a large-scale empirical evaluation of 260 models, revealing that the reliability of label-free metrics is heavily moderated by architecture class and training objective, factors often obscured in aggregate analyses.

Key Results

Intrinsic Dimension (ID) is the Most Reliable Predictor: Among all metrics considered, ID (specifically the TwoNN estimator) shows the most consistent negative correlation with test accuracy across architectures and training objectives. Lower intrinsic dimensionality generally correlates with better generalization. However, its reliability degrades for tasks lacking clear coarse object-category structures (e.g., scene recognition and remote sensing).
Architecture and Objective Dependence:
- Spectral Metrics: Metrics like RankMe and NE Sum are not uniformly reliable. RankMe shows strong correlations when grouped by architecture but weak aggregate correlations. NE Sum is a reliable predictor specifically for SSL models (associated with feature whitening) but performs poorly for supervised models.
- Relational Metrics: Self-Cluster and DSE are highly correlated (negatively) and are reliable predictors for SSL models, reflecting the uniform distribution of representations encouraged by SSL objectives. However, they are largely uninformative for supervised models and specific architectures like ViT (where values often collapse to extremes).
- Condition Number ( $\kappa$ ): Its correlation with OLS accuracy is largely an artifact of the linear solver's numerical conditioning rather than a true measure of representation quality, as the correlation vanishes when using a KNN probe.
Metric Instability: $\alpha$ -ReQ is found to be unstable, with results heavily dependent on the fitting range of the eigenspectrum, contradicting the hypothesis that $\alpha \approx 1$ universally indicates high-quality representations.

Significance and Claims

The paper claims to provide a clearer understanding of what label-free metrics measure, when they are reliable, and how to interpret them in practice. The primary significance lies in the finding that no single label-free metric serves as a universal proxy for downstream performance across all configurations.

The authors conclude that:

Context is Critical: The reliability of a metric is strongly influenced by the model's architecture and training objective. Metrics should not be used to rank models from different architectural classes or training paradigms jointly without caution.
ID as a Robust Baseline: Intrinsic Dimension emerges as the most broadly reliable metric, particularly for object classification tasks, though it is not perfect for all task types.
Limitations of Current Metrics: Spectral and relational metrics capture specific geometric properties (e.g., variance distribution or uniformity) that are only indicative of quality under specific training regimes (e.g., SSL-induced whitening).
Future Directions: The authors suggest that combining complementary metrics (e.g., as in CLID) may be a promising direction, but emphasize that current label-free metrics must be used with caution and an understanding of their specific regimes of validity.

The study does not propose a new metric but rather a framework for interpreting existing ones, warning against the blind application of these metrics as standalone indicators of model quality without considering the underlying model configuration.