💻 computer science

Are Feature Selection and Feature Attribution the Same? A Comparative Survey -- Extended Version

This paper investigates the relationship between feature selection and feature attribution methods through a comprehensive survey and unified evaluation framework, aiming to clarify their similarities, differences, and unique strengths to bridge the gap between these two research fields.

Original authors: Muhammad Rajabinasab, Arthur Zimek

Published 2026-08-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Rajabinasab, Arthur Zimek

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern data science, where computers sift through mountains of information to find patterns, two distinct tools have emerged to help humans understand what the machines are doing. The first tool, known as feature selection, acts like a strict editor. Its job is to look at a raw dataset and decide which pieces of information are truly necessary to solve a problem, discarding the rest to keep the model simple and efficient. It asks, "What do we need to know?" The second tool, called feature attribution, works differently. It does not try to simplify the data before the computer learns; instead, it interrogates a model that has already been trained. It asks, "What did the model actually use to make that specific decision?" This second approach has become vital in the field of explainable artificial intelligence, where people demand to know the reasoning behind a computer's choices, from medical diagnoses to loan approvals. For years, these two groups of methods have operated in separate circles, with statisticians focusing on the first and artificial intelligence researchers focusing on the second, rarely comparing their results directly.

A team of researchers at the University of Southern Denmark decided to bridge this gap by treating these two approaches as two sides of the same coin. They gathered twelve of the most important methods from both families—six from the traditional feature selection side and six from the modern feature attribution side—and put them through a rigorous, head-to-head test. They ran these methods on twenty-three different high-dimensional datasets, which included complex information from fields like genomics and image recognition, where the number of data points often far exceeds the number of examples available. The researchers did not just look at how well the methods performed; they also examined how similar the methods were to each other, how stable their choices remained when the data changed slightly, and how much time and computing power each method required. They tested these methods in two different scenarios: one where they allowed the selection of a broad range of features, and another where they forced the methods to choose only a tiny fraction of the available data, simulating extreme conditions where only the most critical information could be kept.

The results revealed a fascinating split in performance that depends entirely on the goal of the task. When the goal was to predict a specific outcome, such as classifying an image or diagnosing a condition, the feature attribution methods consistently outperformed the traditional feature selection methods. The methods that asked "what did the model use" were better at identifying the features that actually drove the computer's success. This suggests that for predictive tasks, understanding the internal logic of a complex model is a more effective guide than simply looking for statistical patterns in the raw data. However, the story changed completely when the researchers moved to unsupervised tasks, where the computer tries to find natural groupings in data without any pre-defined answers. In these cases, the traditional feature selection methods proved superior. Because these methods are not biased by a specific prediction goal, they were better at preserving the natural structure and shape of the data, keeping the information intact in a way that the model-focused methods sometimes missed.

The study also uncovered a significant trade-off between accuracy and speed. The traditional feature selection methods were incredibly fast, scaling efficiently even as the amount of data grew massive. In contrast, the feature attribution methods, while more accurate for prediction, were computationally expensive. To generate a global view of importance, these methods had to calculate the influence of every single feature for every single data point, a process that became prohibitively slow on large datasets. The researchers found that the two approaches were not as different as they seemed; in many cases, the features selected by the fast, traditional methods were nearly identical to those identified by the slow, attribution-based methods. This convergence suggests that while the two fields ask different questions, they often arrive at the same answer.

Ultimately, the paper argues that the best path forward is not to choose one method over the other, but to use them together in a specific sequence. The researchers propose a hybrid pipeline where the fast, traditional methods are used first to quickly filter out the vast majority of useless data, reducing the problem to a manageable size. Then, the slower, more precise feature attribution methods can be applied to this smaller set to identify the final, most critical features that the model relies on. This approach combines the speed and scalability of the old guard with the high-fidelity accuracy of the new, offering a practical solution for handling the massive, complex datasets that define modern artificial intelligence. The work confirms that while the philosophy behind these methods differs—one seeking simplicity, the other seeking accountability—their practical application is most powerful when they are brought together.

Technical Summary: Are Feature Selection and Feature Attribution the Same? A Comparative Survey

Problem Statement

Feature Selection (FS) and Feature Attribution (FA) are two distinct research fields that often appear functionally similar yet operate under different philosophical paradigms. FS, rooted in the principle of parsimony (Occam's Razor), aims to identify a minimal subset of relevant features to reduce dimensionality and prevent the curse of dimensionality before model training. FA, emerging from the Explainable AI (XAI) movement, seeks to attribute a model's specific prediction to its input features to ensure accountability and transparency, often asking "what did the model actually use?" rather than "what is objectively relevant?"

Despite their technical similarities—both producing feature importance scores and rankings—there is a significant gap in the literature regarding a comprehensive theoretical and empirical comparison between these two groups. Existing surveys typically treat them in isolation, focusing either on FS advances or XAI methods, without addressing whether their underlying mechanisms, evaluation criteria, and performance characteristics are fundamentally different or merely variations of the same process applied at different granularities.

Methodology

The authors conduct a comparative survey and empirical evaluation involving 12 fundamental methods: 6 from the Feature Selection family (Mutual Information, LASSO, Fisher Score, Recursive Feature Elimination, White-box Feature Selection, ReliefF) and 6 from the Feature Attribution family (SHAP, LIME, Permutation, Feature Ablation, Kernel SHAP, Saabas). Integrated Gradients and Sobol Indices were explicitly excluded from the experimental evaluation due to limitations (differentiable model requirements and computational expense, respectively).

Experimental Setup:

Datasets: 23 high-dimensional benchmark datasets from the scikit-feature repository, spanning genomics, proteomics, and image recognition, where the number of features often exceeds the number of instances.
Evaluation Framework: The study employs a unified evaluation framework where FA methods are aggregated (via mean absolute averaging) to produce global importance scores, allowing them to be evaluated using standard FS metrics.
Scenarios: Two experimental scenarios were defined:
1. General Feature Selection: Selecting 5% to 100% of features.
2. Extreme Dimensionality Reduction: Selecting 0.5% to 10% of features to test effectiveness in high-stakes reduction.
Metrics:
- Supervised: Classification Accuracy (ACC), Area Under the ROC Curve (AUC).
- Unsupervised: Clustering Accuracy (CLSACC), Normalized Mutual Information (NMI).
- Model-Agnostic/Geometric: Average Angle Difference (AAD).
- Stability & Similarity: Feature Selection Similarity (intersection of top- $k$ features), Stability metrics, and Computational Scalability.
Ranking Analysis: Results were analyzed using both Standard Rank statistics and Magnitude-Aware Rank Statistics (MARS) to account for performance gaps.

Key Contributions

Unified Taxonomy and Review: The paper provides a detailed review of the theoretical backgrounds and operational mechanisms of both FS and FA methods, situating them within a coherent taxonomy (Filter, Wrapper, Embedded vs. Surrogate, Perturbation, Path-based).
Evaluation Framework Bridging: It proposes a method to evaluate FA methods using standard FS metrics by aggregating local attributions into global scores, thereby enabling a direct, apples-to-apples comparison.
Comprehensive Empirical Comparison: The study presents extensive experimental results across 23 datasets, analyzing performance trajectories, similarity heatmaps, and scalability.
Theoretical Discussion: It addresses the philosophical tension between "parsimony" (simplification) and "interrogability" (accountability), analyzing how these philosophies manifest in operational mechanisms and evaluation outcomes.

Key Results

Supervised Tasks (ACC, AUC): Feature Attribution methods (specifically Tree SHAP, Saabas, and Ablation) consistently outperformed traditional Feature Selection filters (like Fisher Score and Mutual Information). The authors attribute this to FA methods capturing high-order interactions and non-linear boundaries learned by the predictive model (Random Forest), effectively optimizing for the specific downstream task.
Unsupervised Tasks (CLSACC, NMI): Traditional Feature Selection methods (particularly Fisher Score and ReliefF) demonstrated superior resilience. Because FA methods are tied to a specific classification objective, they may discard features crucial for the global manifold structure required for clustering. FS filters, being model-agnostic, better preserve the data's natural groupings.
Geometric Preservation (AAD): A paradox was observed where methods performing best in supervised tasks often performed worst in preserving the raw geometric properties of the data (measured by AAD). This suggests that achieving high predictive accuracy often requires "breaking" the original data geometry to find a separable manifold.
Similarity and Convergence: High similarity was observed between LASSO (FS) and Permutation (FA), suggesting that for many high-dimensional problems, the features necessary for a sparse model are the same as those most impactful when removed from a complex model.
Scalability: Statistical FS filters (e.g., Fisher, MI) scale linearly and are computationally efficient. In contrast, FA methods, which require computing local attributions for every instance to derive global scores, exhibit significant computational bottlenecks in high-dimensional settings.
LIME Performance: LIME consistently performed poorly in global evaluations, attributed to its extremely local perspective and reliance on surrogate models that struggle to generalize global importance.

Significance and Claims

The paper claims to bridge the gap between the fields of Feature Selection and Feature Attribution, demonstrating that while they share operational cores (perturb-and-observe logic), they serve distinct purposes based on the task context.

Task-Dependent Superiority: The authors conclude that there is no single "best" method. Feature Attribution is superior for task-specific optimization (supervised prediction) because it interrogates the model's internal logic. Feature Selection is superior for data exploration and unsupervised tasks because it preserves the data's intrinsic structure without model bias.
Hybrid Pipeline Proposal: Based on scalability and performance findings, the paper suggests a hybrid pipeline for high-dimensional data mining: first, use efficient FS filters to reduce dimensionality (removing noise), and second, apply FA methods to the reduced set to select the final subset based on model accountability.
Epistemological Agreement: The study highlights that despite different philosophical starting points, the two fields often converge on the same critical features, validating the transferability of insights between them.

The authors maintain modesty regarding their claims, noting that their evaluation relied heavily on Random Forest as the backend model and that alternative aggregation strategies for FA methods might yield different results. They position this work as a foundational step toward integrating statistical parsimony with algorithmic accountability in future high-dimensional machine learning pipelines.