🤖 machine learning

Worse than Random: The Importance of a Baseline for Unsupervised Feature Selection

The paper argues that random feature selection should serve as a mandatory baseline for evaluating unsupervised feature selection methods, as many state-of-the-art approaches are empirically shown to be outperformed by random selection in both performance and efficiency.

Original authors: Muhammad Rajabinasab, Michael E. Houle, Oussama Chelly, Arthur Zimek

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammad Rajabinasab, Michael E. Houle, Oussama Chelly, Arthur Zimek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Are We Actually Improving Anything?

Imagine you are a chef trying to create the perfect soup. Every year, new chefs (researchers) propose fancy new recipes (algorithms) for picking the best ingredients (features) from a giant pantry. They claim their new method is "state-of-the-art" and will make the soup taste amazing.

But here is the catch: Nobody is checking if their fancy recipe is actually better than just grabbing a handful of ingredients at random.

In the world of machine learning, specifically "unsupervised feature selection," researchers try to find the most important data points without having a "teacher" (labels) to tell them what is right or wrong. The paper argues that for years, these researchers have been comparing their complex methods only against other complex methods. They never compared them to the simplest possible approach: Random Chance.

The Core Idea: The "Random Grab" Baseline

The authors propose a simple rule: Before you claim your new method is great, it must beat a monkey throwing darts at a dartboard.

The Complex Method: A sophisticated AI that analyzes data, calculates correlations, and uses heavy math to decide which features to keep. It takes a long time and costs a lot of computing power.
The Baseline (Random Feature Selection): Imagine you have 1,000 ingredients. You close your eyes, spin a wheel, and pick 100 of them. That's it. No math, no thinking, just luck.

The paper's shocking discovery is that many of the most advanced, expensive, and complex AI methods are actually performing worse than just picking ingredients randomly.

The Experiment: The Taste Test

The researchers ran a massive "taste test" (experiments) using 23 different high-dimensional datasets (think of these as very complex, messy pantries with thousands of ingredients).

They tested:

Old-school methods: Simple math like checking how much ingredients vary (Variance) or how they relate to each other (Correlation).
Brand-new "State-of-the-Art" methods: Complex neural networks and graph-learning frameworks that take hours to run.
The Random Baseline: Just picking features at random.

The Results:

Speed: The random method was the fastest by far. It took seconds. The fancy new methods took hours or even crashed because they were too heavy.
Performance: When they used the selected ingredients to make a soup (run a classification or clustering task), the random method often tasted just as good, or even better, than the fancy methods.
The "Z-Score" Reality Check: The authors used a statistical tool (Z-score) to see how far off the mark the fancy methods were. They found that most advanced methods were actually below the random baseline. In other words, they were doing a worse job than luck.

Why Did This Happen?

The paper suggests a few reasons why this "Worse than Random" phenomenon exists:

The "Noise" Problem: In very high-dimensional data (like images or gene data), there is so much "noise" (irrelevant information) that it's hard to find the signal. Sometimes, removing 90% of the data randomly actually helps because you accidentally remove the noise.
Missing the Goalpost: Researchers have been so focused on comparing Method A vs. Method B that they forgot to ask: "Is either of these actually better than doing nothing?"
False Confidence: Because the datasets used for testing are often small or simple, complex models can look good just by overfitting (memorizing the test data) rather than actually learning something useful.

The Takeaway: Don't Build a Ferrari to Go to the Grocery Store

The main conclusion of the paper is a call to action for the scientific community:

Stop inventing complex, expensive machines if they don't beat a bicycle.

If a new unsupervised feature selection method cannot consistently beat a random selection of features, it is not adding value. It is just adding cost and complexity. The authors argue that Random Feature Selection should be the mandatory "baseline" (the minimum standard) for all future research.

If your method is slower and costs more than random: It's probably not worth it.
If your method is only slightly better than random: It might not be worth the effort.
If your method is significantly better: Then you have a breakthrough.

Summary in a Nutshell

The paper is a reality check. It tells the machine learning community: "You are building incredibly complex tools to solve a problem, but you haven't proven they work better than just guessing. Before you publish your next 'revolutionary' algorithm, make sure it can beat a coin flip. If it can't, you're just making the soup more complicated, not tastier."

Technical Summary: Worse than Random: The Importance of a Baseline for Unsupervised Feature Selection

Problem Statement
The paper addresses a critical gap in the empirical evaluation of unsupervised feature selection (UFS) methods. Despite the annual proposal of numerous novel UFS algorithms, their evaluation is often limited to comparisons against other existing methods on specific datasets, lacking a standardized, established baseline. The authors argue that without such a baseline, it is difficult to determine the actual value added by new methods or the effectiveness of their underlying approaches. Furthermore, performance comparisons are not transitive across different dataset collections, making it impossible to justify whether a complex, opaque model provides meaningful gains over trivial solutions or if observed improvements are merely the result of random coincidence or simplistic dataset structures.

Methodology
The authors propose Random Feature Selection (RFS) as a mandatory baseline for evaluating unsupervised feature selection methods. RFS involves assigning feature importance scores drawn from a uniform random distribution and selecting the top $k$ features based on these scores. This approach is:

Unsupervised: It requires no labels.
Efficient: It involves no expensive computational steps beyond generating random values.
Logical: In very high-dimensional spaces where redundancy is high, even random removal of features is expected to yield acceptable overall performance.

To validate this proposal, the authors conducted extensive experiments on 23 high-dimensional benchmark datasets (ranging from 50 to 7,000 instances and 325 to 22,283 features). They compared traditional methods (Variance-based, Correlation-based, Laplacian Score, MCFS) and recent state-of-the-art (SOTA) methods (SCFS, SOGFS, LLSRFS, VCDFS, RAFG) against the RFS baseline.

The evaluation utilized a dual-experiment design:

Selecting 5% to 100% of features (step size 5%).
Selecting 0.5% to 10% of features (step size 0.5%) to simulate extreme dimensionality reduction.

Performance was measured using:

Supervised metrics: Accuracy (ACC) and Area Under the Curve (AUC) via Random Forest classifiers with 5-fold cross-validation.
Unsupervised metrics: Clustering Accuracy (CLSACC) and Normalized Mutual Information (NMI) via k-means clustering.
Aggregate metrics: Feature Selection Dynamic Evaluation Metric (FSDEM) to assess the overall process quality, and Z-scores to measure the magnitude of difference relative to the RFS distribution.

Key Results
The empirical analysis yielded several striking findings:

Efficiency: RFS is the most efficient method by a large margin, as it requires no significant computational operations. In contrast, recent SOTA methods (e.g., SOGFS, LLSRFS) are computationally expensive, with runtimes often capped or exceeding one hour on large datasets.
Performance: Contrary to expectations, many SOTA methods failed to outperform the random baseline. In numerous cases, complex algorithms performed worse than RFS in both supervised and unsupervised downstream tasks.
Statistical Significance: Critical difference diagrams (based on Wilcoxon-Holm tests) revealed that while some methods (like SCFS for supervised tasks or Laplacian Score for unsupervised tasks) occasionally ranked slightly higher, the differences were often statistically insignificant. The overall rank of most methods was very close to, or worse than, the random baseline.
Z-Score Analysis: Many methods exhibited significantly negative Z-scores relative to the RFS distribution, indicating consistent underperformance compared to random chance.

Significance and Claims
The paper claims that the field of unsupervised feature selection suffers from a "critical lack of a baseline." The authors emphasize that:

RFS is a Necessary Baseline: It serves as a strict requirement for the development of novel UFS methods. A new method must demonstrate a significant improvement over random selection to justify its computational complexity and cost.
Current Literature is Flawed: The fact that many SOTA methods perform worse than or equal to random selection suggests that their proposed gains may not be meaningful. The reliance on downstream task performance (e.g., classifier accuracy) without a baseline has masked the poor quality of the feature selection process itself.
Justification of Complexity: If a complex model only marginally outperforms a trivial baseline (or fails to do so), its complexity is unjustified. The authors argue that RFS provides a "consistent improvement" benchmark that ensures proposed methods offer genuine value rather than random coincidence.

In conclusion, the paper does not propose a new feature selection algorithm but rather a new standard for evaluation. It asserts that random feature selection should be the default reference point to ensure that future research in unsupervised feature selection yields methods that are truly effective and computationally justified.