Technical Summary: Objective-Induced Bias and Search Dynamics in Multiobjective Unsupervised Feature Selection

1. Problem Definition

Unsupervised Feature Selection (UFS) aims to identify compact feature subsets that preserve dataset information without relying on target labels. While often formulated as a multiobjective optimisation problem (MOFS) to jointly optimise subset quality and size, the efficacy of this approach is critically dependent on design choices that remain underexplored. Specifically, the behaviour of MOFS formulations is sensitive to:

• The choice of the evaluation objective (e.g., clustering quality vs. reconstruction loss).
• The direction of subset-size regularisation (minimisation vs. maximisation).
• The initialisation strategy of the population in evolutionary algorithms.

Without explicit control, these factors can lead to degenerate solutions, such as trivial low-cardinality subsets that fail to capture informative structures, or search dynamics that fail to explore relevant regions of the Pareto front within a limited evaluation budget.

2. Methodology

2.1 Experimental Framework

The authors propose a controlled experimental framework using a synthetic dataset with a known feature taxonomy. This dataset includes:

• Informative Features: Designed to separate classes with added noise.
• Redundant Features: Linear and non-linear combinations of informative features.
• Noise Features: Gaussian noise and structured noise (which possess internal structure but are unrelated to the target).
• Sweep Features: Informative features progressively perturbed to test signal prioritisation.

This setup allows for direct inspection of the composition of selected subsets (informative, redundant, irrelevant) rather than relying solely on objective values.

2.2 Multiobjective Formulations

The study compares six MOFS formulations by combining three evaluation objectives ($f_1$) with two subset-size regularisation directions ($f_2$):

1. Evaluation Objectives:
   • Silhouette Score: A clustering quality metric measuring intra-cluster cohesion and inter-cluster separation.
   • Classification Accuracy: A supervised baseline (Random Forest) used for reference.
   • PCA Reconstruction Loss (Proposed): A novel unsupervised objective that minimises the Mean Square Error between the projection of the full dataset onto its principal components and the reconstruction of that projection using only the selected feature subset via linear regression. This explicitly addresses feature redundancy by evaluating collective information rather than individual variance contribution.
2. Regularisation ($f_2$):
   • Minimisation of subset size ($|x|$).
   • Maximisation of subset size ($-|x|$).

2.3 Search Dynamics and Initialisation

The optimisation is performed using the SMS-EMOA algorithm. The study investigates three initialisation strategies to determine their impact on search dynamics:

• Binary Random Sampling: Independent Bernoulli trials for feature selection.
• Segmented Sampling: The population is divided into sub-populations with different selection probabilities (e.g., $p \in \{0.25, 0.5, 0.75\}$) to ensure diversity in cardinality.
• Fixed Cardinality Sampling: All individuals start with the same number of selected features.

3. Key Contributions

1. Analysis of Design Choices: The study demonstrates how subset-size regularisation direction and initial population sampling strategies shape the structure of the Pareto front and determine which regions of the search space are explored under limited computational budgets.
2. Feature Composition Analysis: Using the synthetic dataset with known feature lineage, the authors characterise how different objectives select feature types (informative vs. redundant vs. irrelevant).
3. Introduction of PCA Loss: The paper introduces PCA reconstruction loss as an unsupervised objective for feature selection. It analyses this objective's behaviour under subset-size regularisation, showing it effectively addresses cardinality bias and aligns with downstream classification performance.

4. Results

4.1 Impact of Regularisation and Initialisation

• Accuracy + Size Minimisation: This standard formulation yields well-distributed Pareto fronts. However, the initialisation strategy significantly affects the search. Counter-intuitively, starting with low-cardinality populations (e.g., single features) often yields better results than starting with larger subsets, as the accuracy objective drives the search toward larger subsets only when performance gains are realized, avoiding the trap of getting stuck in high-cardinality local optima.
• Silhouette + Size Minimisation: This combination induces a strong bias toward trivial, low-cardinality solutions. The silhouette score is inherently biased toward small subsets in high-dimensional spaces (the "curse of dimensionality"), where small subsets can artificially inflate scores by isolating chance clusters. Even with appropriate regularisation, this formulation fails to capture informative structures and collapses toward the origin of the search space.

4.2 Composition of Solutions

• Silhouette-Based Formulations: Even when paired with size maximisation (to counteract the bias), silhouette-based objectives tend to select redundant and irrelevant features. The resulting Pareto fronts are dominated by solutions that do not achieve high classification accuracy, indicating that silhouette score is a poor proxy for predictive performance in this context.
• PCA Loss-Based Formulations: The proposed PCA loss objective, when paired with size minimisation, successfully retains a substantial proportion of informative features while discarding pure noise. The method effectively approximates near-optimal feature subsets. Solutions on the Pareto front show a clear trade-off between subset size and reconstruction loss, with performance levels comparable to those obtained by directly optimising supervised accuracy.

4.3 Comparative Performance

When evaluated on a held-out test set, the PCA loss objective produced compact subsets with test accuracy comparable to the supervised accuracy baseline. In contrast, silhouette-based formulations consistently underperformed, regardless of the regularisation direction.

5. Significance and Claims

The paper claims that objective design is central to effective multiobjective unsupervised feature selection. The study highlights that:

• Bias is Objective-Induced: The choice of evaluation objective (e.g., clustering metrics vs. reconstruction loss) fundamentally dictates the search dynamics and the quality of the resulting Pareto front.
• Clustering Metrics are Flawed Proxies: Silhouette scores and similar clustering metrics are prone to inducing biases toward trivial, low-cardinality solutions and are weak proxies for predictive performance.
• PCA Loss is a Viable Alternative: The proposed PCA reconstruction loss offers a stable, unsupervised signal that balances subset compactness with predictive performance, effectively handling feature redundancy without requiring target labels.

The authors conclude that while their findings are robust on the controlled synthetic dataset, future work is required to verify these results on real-world datasets and explore more complex latent representations using non-linear reconstruction models.