🤖 machine learning

Radial Basis Function Networks as Projection Heads in Self-Supervised Learning

This paper proposes replacing the standard MLP projection head in self-supervised learning with a Radial Basis Function Network (RBFN) to enable label-free assessment of representation quality via a novel Scale-Normalized Separation (SNS) metric, while maintaining competitive performance across multiple architectures and datasets.

Original authors: Andreas Schliebitz, Heiko Tapken, Martin Atzmueller

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andreas Schliebitz, Heiko Tapken, Martin Atzmueller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Useless" Head

Imagine you are training a robot to recognize different types of animals. To do this, you give the robot a "brain" (called a backbone) that looks at pictures and figures out the important features.

In current self-supervised learning (where the robot learns without a teacher telling it the answers), researchers add a small "helper hat" on top of the brain. This hat is called a projection head. Its job is to help the brain learn better during the training phase.

The Problem: Once the robot finishes learning, scientists usually throw this "helper hat" away. They then have to spend a lot of time and money teaching the brain a new, simple way to sort the animals just to see how good the brain actually is. The authors of this paper say: "Wait a minute! Throwing away the hat is wasteful. Why not keep it and use it to check if the brain is doing a good job?"

The Solution: A "Smart Hat" Instead of a "Dumb Hat"

Usually, this helper hat is a standard, slightly complex machine (an MLP) that is hard to understand. You can't easily tell why it made a decision.

The authors propose swapping this standard hat for a Radial Basis Function Network (RBFN).

The Analogy: Think of the standard hat as a black box that just gives you a number. The new RBFN hat is like a map with specific landmarks.
How it works: Instead of just crunching numbers, the RBFN learns specific "centers" (landmarks) and "shapes" (how wide the influence of each landmark is).
The Benefit: Because these landmarks and shapes are so clear and interpretable, you can look at the hat itself and say, "Ah, the landmarks are well-spaced out, so the brain must be doing a great job organizing the data." You don't need to throw the hat away or hire a new teacher to check the work.

The New Tool: "SNS" (The Quality Score)

To prove their idea, the authors invented a new way to measure quality called Scale-Normalized Separation (SNS).

The Metaphor: Imagine you are organizing a party where guests (data points) need to sit at tables (clusters).
- If everyone is crammed into one corner, the party is a mess (bad quality).
- If everyone is spread out perfectly, with just the right amount of space between tables, the party is great (good quality).
How SNS works: It looks at the "landmarks" (centers) the RBFN hat learned. It checks if the distance between these landmarks matches the "size" of the tables (the shape parameters).
The Result: If the landmarks are perfectly spaced out, the SNS score is high. The authors found that a high SNS score almost always means the robot's brain is actually very good at recognizing things, even without looking at any labeled answers.

What They Tested

The researchers tested this idea on five different popular robot-learning systems (like MoCo, SimCLR, and BYOL) and four different sets of image data (including a new dataset they created based on Google's Open Images).

They asked two main questions:

Does the new "Smart Hat" work as well as the old "Dumb Hat"?
- Answer: Yes. The robots learned just as well with the RBFN hat. In some cases, it was even slightly better. It's a perfect "drop-in" replacement.
Can we trust the SNS score to tell us if the robot is smart?
- Answer: Yes. They compared the SNS score against the traditional, expensive method (training a new classifier on labeled data). They found a very strong connection: when the SNS score was high, the robot's actual performance was also high.

The Recommendations

Based on their experiments, the authors suggest:

Use 3 Layers: Build the RBFN hat with three layers of "landmarks" instead of the usual two. This seems to work best.
Use Gaussian Shapes: Use a specific mathematical shape (Gaussian) for the landmarks, as it worked better than other shapes.
Don't Worry About Normalization: You don't need to do extra math to "normalize" the hat; it works fine as is.

Summary

The paper argues that we are wasting resources by throwing away the "helper hat" after training. By swapping the standard hat for a Radial Basis Function Network, we get a tool that is just as good at helping the robot learn, but also acts as a built-in quality report card. We can look at the hat's internal map (the centers and shapes) to instantly know if the robot is learning well, saving us time and money.

Technical Summary: Radial Basis Function Networks as Projection Heads in Self-Supervised Learning

Problem Statement
In standard Self-Supervised Learning (SSL) pipelines, a backbone encoder (e.g., ResNet, ViT) is paired with a shallow Multilayer Perceptron (MLP) projection head. While the MLP head is crucial for training stability and representation quality, it is conventionally discarded after pre-training. The backbone's quality is then assessed via a costly linear probing procedure on labeled data. The authors argue that discarding the trained projector is computationally and information-theoretically wasteful. Furthermore, the reliance on labeled data for evaluation creates a bottleneck, as it prevents the assessment of representation quality in truly unsupervised or label-scarce scenarios.

Methodology
The paper proposes replacing the standard MLP projection head with a Radial Basis Function Network (RBFN) head. This architectural change serves two purposes: maintaining competitive representation learning performance and enabling label-free quality assessment.

RBFN Projection Head:
- The authors implement a drop-in replacement for MLP heads compatible with the LightlySSL framework.
- The architecture consists of linear layers followed by RBF layers. The non-linearity is provided by radial basis functions (specifically Gaussian, inverse multiquadric, multiquadric, and thin-plate spline) rather than ReLU.
- The RBF layer learns a set of kernel centers ( $c_i$ ) and shape parameters ( $\varepsilon_i$ ). The output is a linear combination of these radial basis functions.
- The authors test this across five canonical SSL architectures: MoCo, SimCLR, BYOL, SwAV, and SimSiam.
Scale-Normalized Separation (SNS) Metric:
- To evaluate representation quality without labels, the authors introduce SNS, a metric derived solely from the learned RBFN parameters ( $c_i$ and $\varepsilon_i$ ).
- SNS quantifies the "separation" of kernel centers relative to their scales. It calculates a Gaussian interaction energy between pairs of centers, normalized by the dimensionality of the space and the sum of their shape parameters.
- The metric is maximized when inter-center distances are well-matched to the kernel scales, theoretically indicating a well-structured clustering of features in the projection space.
Experimental Setup:
- Experiments were conducted on four datasets: ImageNet-100 and three subsets of Google's Open Images V7 (10, 30, and 50 classes).
- A grid search was performed over RBFN hyperparameters, including the number of kernels, depth of the projector, normalization, and the type of radial basis function.
- Performance was benchmarked against standard MLP heads using linear probing (Logistic Regression) metrics (Accuracy, Precision, Recall, F1).
- Correlation analysis (Pearson and Spearman) was performed between the SNS scores and the standard linear probing metrics.

Key Contributions

RBFN as a Drop-in Replacement: The authors demonstrate that RBFN projection heads are competitive with standard MLP projectors across five major SSL architectures, serving as a direct replacement without significant performance degradation.
SNS Metric: They introduce a novel, label-free quality metric (SNS) that correlates strongly with downstream classification performance. This allows for the assessment of backbone representation quality using only the parameters learned during self-supervised training.
Hyperparameter Guidelines: Through SHAP analysis, the paper provides empirical recommendations for constructing RBFN heads:
- Use Gaussian radial basis functions for activation.
- Construct heads with three layers (linear-RBF-linear-RBF-linear) rather than the standard two.
- Set the number of kernels to match the output dimension (e.g., 128).
- Avoid normalization of the RBFN architecture in most cases.
New Dataset Resource: The authors release a PyTorch-compatible implementation of Open Images V7 subsets (10, 30, 50, 70, 100 classes) with disjoint classes from ImageNet-1K to facilitate reproducible research.

Results

Performance Parity: RBFN heads achieved performance comparable to default MLP heads. On ImageNet-100, RBFN heads slightly outperformed MLPs on SimCLR (+4% accuracy) and matched them on other datasets. On Open Images subsets, RBFNs showed a minor deficit (1–3%) on larger subsets but outperformed MLPs by 2–4% when used with BYOL.
Metric Correlation: The SNS metric exhibited strong to very strong positive correlations (Pearson $r$ ranging from 0.79 to 0.93) with established logistic regression metrics (Accuracy, F1, etc.). The p-values for these correlations were extremely low, supporting the hypothesis that SNS is a reliable proxy for representation quality.
Hyperparameter Impact: The choice of RBF function (Gaussian) and projector depth (3 layers) had the most significant impact on performance. The number of kernels and normalization showed negligible effects within the tested ranges.

Significance and Claims
The paper claims that replacing MLP projectors with RBFNs offers a dual benefit: it maintains the high performance required for SSL while unlocking the ability to evaluate representation quality without labeled data. By interpreting the learned centers and shapes of the RBFN, researchers can gauge the quality of the backbone's representations via the SNS metric. The authors position this as a move away from "computationally wasteful" discarding of projectors toward a more interpretable and efficient evaluation pipeline. They conclude that a trained RBFN projector can act as a reliable proxy for backbone quality, potentially reducing the dependency on labeled test sets for model selection in self-supervised learning.