📊 statistics

Is Supervised Learning Really That Different from Unsupervised?

The paper proposes a novel two-stage framework that enables various supervised learning algorithms to achieve performance comparable to their standard counterparts by selecting model parameters entirely through unsupervised methods, thereby suggesting that the fundamental distinction between supervised and unsupervised learning may be less significant than previously thought.

Original authors: Oskar Allerbo, Thomas B. Schön

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Oskar Allerbo, Thomas B. Schön

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize different types of fruit. Usually, to do this, you show the robot thousands of pictures of apples, bananas, and oranges, and you tell it, "This is an apple," "This is a banana." This is Supervised Learning: the robot learns by looking at the pictures (the data) and the labels (the answers).

This paper asks a surprising question: What if we taught the robot how to sort the fruit without ever telling it what the fruit actually is?

The authors, Oskar Allerbo and Thomas B. Schön, argue that supervised learning isn't as magical or dependent on "answers" as we think. They propose a two-step process that turns a supervised task into an unsupervised one.

The Two-Step "Sort First, Label Later" Method

Think of the training data as a big box of mixed-up Lego bricks.

Step 1: The Unsupervised Sort (The "Shape" Phase)
Instead of looking at the instruction manual (the labels), the robot looks only at the shape, color, and size of the bricks. It uses an unsupervised algorithm to group them.

It might say, "All the red, round bricks go in this pile."
"All the blue, long bricks go in that pile."
"All the yellow, square bricks go in that other pile."

Crucially, at this stage, the robot does not know that the red pile is "apples" or the blue pile is "bananas." It has just learned the structure of the data. It has figured out how the data points relate to each other.

Step 2: The Label Reveal (The "Name" Phase)
Once the robot has perfectly sorted the bricks into four distinct piles, then you reveal the labels.

You say, "Okay, the red pile is Apples."
"The blue pile is Bananas."

The robot simply assigns the name "Apple" to the red pile and "Banana" to the blue pile. It doesn't need to re-sort the bricks or change how it grouped them. The "smart part" (figuring out the groups) was already done without the labels.

The Secret Sauce: A New "Variance" Rule

You might ask: "How does the robot know how to sort the bricks in Step 1 if it can't check its work against the answers?"

Usually, to check if a sorting method is good, you use a technique called Cross-Validation, where you hide some answers, make a guess, and see if you were right. But that requires knowing the answers (the labels) in the first place.

The authors invented a new rule called MSV (Matching Sample Variances).

The Analogy: Imagine you are trying to tune a radio to find a clear station. Usually, you listen to the music to know if you are tuned right. But here, the authors say: "Don't listen to the music (the labels). Just listen to the static (the noise)."
They found that if the robot sorts the data in a way that the "spread" or "variety" of its guesses matches the natural "spread" of the data itself, it's likely doing a good job.
They proved mathematically that you can calculate this "spread" without ever looking at the labels. It's like tuning a radio by matching the static to the background noise of the room, rather than listening for a song.

What They Tested

They took this idea and applied it to many different types of "robots" (algorithms):

Linear Regression: Predicting numbers (like house prices).
Neural Networks: The complex AI brains used for image recognition.
Random Forests and K-Nearest Neighbors: Other popular sorting methods.

They trained these models on real-world data (like predicting CPU run times or identifying superconductors) and on synthetic data.

The Result:
The models trained without seeing the labels performed almost exactly as well as the models trained with the labels.

In some cases, the "label-free" models were just as accurate.
In other cases, they were slightly better or slightly worse, but never "random guessing" (unless they used the wrong tuning rule, which they proved happens with older methods).

The Big Takeaway

The paper concludes that the difference between "Supervised" (learning with answers) and "Unsupervised" (learning without answers) is less fundamental than we thought.

The Core Idea:
The hard part of learning isn't knowing the answer; it's figuring out the structure of the data. Once you understand the structure (the "shape" of the data), the labels are just a simple tag you can slap on at the very end.

What This Paper Does Not Say

It does not say you should stop using labels in real life. The authors admit their method is computationally heavy (it takes a lot of computer power) and that using labels is still the standard, most efficient way to train models.
It does not claim this is a magic bullet for every problem. It works well for specific types of models (like those that can be described as "smoothers"), but the authors are careful to say this is a theoretical demonstration of a concept, not a new industrial standard.

In short: You can build the skeleton of a house without knowing which room will be the kitchen or the bedroom. Once the walls are up, you can paint the signs on the doors. The paper proves that for many AI models, building the skeleton is the hard part, and the signs are easy to add later.

Technical Summary: Supervised Learning as a Two-Stage Unsupervised Procedure

Problem Statement

The paper addresses a fundamental question in machine learning: Are the output labels ( $y$ ) strictly necessary for training supervised models, or can the model structure be learned entirely in an unsupervised manner? While unsupervised learning operates without access to $y$ , supervised learning traditionally relies on $y$ to optimize model parameters. The authors investigate whether supervised learning can be decomposed into a two-stage procedure where (1) all model parameters (specifically the "smoother" or representation of the input space) are selected without access to $y$ , and (2) the outputs $y$ are added only after the parameters are fixed to generate predictions.

Methodology

1. The Smoother Framework

The core theoretical contribution is the formulation of supervised models as linear smoothers. A model is expressed as:
$\hat{f}^* = s^{*\top} y$
where $\hat{f}^*$ is the prediction for a new covariate $x^*$ , and $s^*$ is a smoother vector (or matrix $S$ for in-sample data) that depends on the training covariates $X$ and potentially model hyperparameters, but not on the outputs $y$ .

For linear models (e.g., Ridge Regression), $s^*$ is derived directly from the closed-form solution.
For non-linear models (e.g., Neural Networks, Random Forests), the authors demonstrate how to express them as smoothers. For Neural Networks, this utilizes the Neural Tangent Kernel (NTK) framework, treating the network as a linear smoother where the smoother matrix is updated iteratively during training but remains independent of the true $y$ if the training labels are replaced.

2. Model Selection Without $y$ (MSV)

To select model hyperparameters (e.g., regularization strength $\lambda$ , kernel bandwidth, or stopping time) without access to $y$ , the authors propose a criterion based on the variance of out-of-sample predictions.

Premise: For a well-generalizing model, the distribution of out-of-sample predictions $\hat{f}^*_v$ should roughly match the distribution of the observed data $y$ .
Criterion: The method minimizes the difference between the sample variance of $y$ and the sample variance of the predictions on a set of validation covariates $X^*_v$ (which are generated from the training distribution but have no associated labels).
y-free Implementation: The objective function involves a term of the form $|y^\top A y|$ . To make this independent of $y$ , the authors replace $y$ with a random vector $y_R$ (or the zero vector). By taking the expectation over $y_R$ , the objective simplifies to minimizing a norm of the matrix $A$ (specifically the trace seminorm or Frobenius norm), denoted as MSV-Tr (Matching of Sample Variances - Trace).
$\text{Minimize } \left| 1 - \frac{1}{n^*_v} \text{Tr}(S^{*\top}_v S^*_v) \right|$
This allows hyperparameter selection using only $X$ and generated $X^*_v$ , completely bypassing the need for $y$ during the selection phase.

3. Theoretical Analysis

The paper provides a bias-variance decomposition analysis for linear ridge regression in the asymptotic limit ( $n, d \to \infty$ ).

Proposition 1: Shows that MSV-Tr selects the model such that the variance of the inferred function equals the variance of the measurement noise ( $\sigma^2_\epsilon$ ).
Theorem 1: For linear ridge regression, the authors derive a closed-form expression for the optimal regularization parameter $\lambda$ selected by MSV-Tr. They prove that the asymptotic out-of-sample risk of this method is bounded relative to the theoretically optimal risk (which requires knowledge of the signal-to-noise ratio). Specifically, for signal-to-noise ratios (SNR) in $[1, 80]$ , the risk ratio is bounded by 2.45.
Theorem 2: Demonstrates that alternative "y-free" methods, such as Generalized Cross-Validation (GCV) and in-sample MSV, fail for ridge regression. They either select infinite regularization (zero training) or zero regularization (overfitting), rendering them useless for parameter selection. Only out-of-sample MSV yields a finite, non-zero optimal regularization.

Key Contributions

Decomposition of Supervised Learning: The paper demonstrates that standard supervised models (linear/kernel ridge regression, smoothing splines, k-NN, random forests, and neural networks) can be trained in a two-stage process where the model structure is learned unsupervised, and labels are only introduced at the prediction stage.
Novel Model Selection Criterion (MSV): Introduction of a model selection criterion based on matching the variance of out-of-sample predictions to the data variance, which can be computed without access to $y$ by replacing $y$ with random noise.
Theoretical Bounds: Establishment of asymptotic risk bounds for linear ridge regression trained via MSV-Tr, showing it performs within a constant factor of the optimal risk.
Empirical Validation: Extensive experiments on synthetic and real-world datasets (MNIST, CIFAR-10, Steel Production, CPU Run Time, Superconductors, Power Consumption) showing that models trained without $y$ (using random or zero labels during the parameter selection phase) achieve performance comparable to standard $y$ -based training.

Results

Performance Parity: In experiments, models trained without $y$ (using MSV for selection) achieved accuracy and $R^2$ scores very close to their standard counterparts trained with $y$ . For example, on MNIST, the standard Neural Network Classifier achieved 0.82 accuracy, while the y-free MSV version achieved 0.75.
Failure of Alternatives: y-free Generalized Cross-Validation (GCV) performed at the level of random guessing (e.g., 0.1 accuracy for 10-class problems), confirming the theoretical findings that GCV cannot be made y-free effectively for these models.
Random Forests: While y-free training worked well for most models, Random Forests showed a larger performance gap compared to y-based training. The authors attribute this to the reliance of decision tree splits on the variance of $y$ , which is lost when $y$ is replaced by random noise.

Significance and Claims

The paper claims that the difference between supervised and unsupervised learning is "less fundamental than it may appear." The results suggest that the primary challenge in supervised learning is finding a good representation of the input space $x$ (the smoother $s^*$ ), which can be achieved without labels. The labels $y$ are necessary only to assign values to these pre-constructed representations.

Modesty of Claims:
The authors explicitly state that they do not suggest excluding $y$ from the training process in practical applications. They acknowledge that:

The smoother formulation scales poorly with data size, particularly for neural networks where the NTK must be recalculated iteratively.
Their method is not intended to outperform existing $y$ -based approaches.
The goal is to demonstrate feasibility and highlight the theoretical parallels between supervised and unsupervised learning, rather than to propose a new standard for industrial deployment.

The work serves as a theoretical and empirical demonstration that model selection and representation learning can be decoupled from label information, challenging the conventional necessity of labels for the entire training pipeline.